search

NINAnor / seatrack-db

0 stars 0 forks source link

R connection times out when getting large datasets #14

Open jenast opened 2 years ago

jenast commented 2 years ago

A possible workaround is to download the data through the terminal (psql). Could perhaps package a psql function in an R function.

jenast commented 2 years ago

The problem seems to be attached to the dplyr/dbplyr subsetting methods. It seems to work better with a standard dbGetQuery.

Redo the getRecordings, pasting together a query and using dbGetQuery. Possibly fetching each 100 million rows at a time using dbFetch, and binding these together.

\copy to local csv and reading that into R looks to be about 2x slower.