Open PatrickRWright opened 4 years ago
I tried a little profiling... It seems to be converting the dates that's primarily causing the lag (at least in the dataset I've looked at)
Specifically, it's a merge in .convert_dates
What's the tool you are using there? Looks super useful.
profvis
- it integrates with RStudio's IDE
see here for tutorial on RStudio's support forum
devtools::load_all()
profvis::profvis(read_secuTrial(path_to_export))
I find it easier to look at the Data tab
We could benchmark the tidyverse
reading functions to see if its worthwhile switching.
In the spirit of structured procrastination I prepared a small benchmark: https://gist.github.com/PatrickRWright/4ed5d4e5b5aed03b7a1aa5b593dd9b64
readr
is faster but its not exactly light speed either.
There is also data.table::fread and vroom::vroom which would be worth looking at. vroom is apparently the fastest, although I've never used it... (screenshot from the vroom readme)
If it were just the numbers I agree. Lets have a look at the dependency consequences it has. The speedup looks pretty impressive though.
Is your feature request related to a problem? Please describe. Very big exports (i.e. tens of thousands of entries) need long to read. Maybe its possible to boost the performance.