Closed andrewdolman closed 3 years ago
Thanks,
This was changed during troubleshooting, because in some (rare) cases, 1000 isn't enough. However, it shouldn't stay at 1e6 because of the overhead. I'll look into better solutions.
On Fri, Feb 21, 2020 at 9:05 AM Andrew Dolman notifications@github.com wrote:
There is a massive performance issue using readLipd in R caused by the argument ",guess_max = 1e6" in the call to read_csv in function read_csv_from_file.
Using the example file from http://nickmckay.github.io/LiPD-utilities/r/index.html, which is only 1.3 MB, uses over 7 GB of RAM.
Setting this to 1e3 fixes the problem.
I would make a pull request but I don't know enough about LiPD to know whether guessing from the first 1000 entries is likely to be good enough.
library(lipdR)
pth <- normalizePath("ODP1098B13.lpd")
dat <- readLipd(pth)
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nickmckay/LiPD-utilities/issues/57?email_source=notifications&email_token=ACXPOZ3ERJJ6L2IHIYDUYMLRD73UPA5CNFSM4KZF3IWKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IPK5IZA, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXPOZ4YEJTAZNS6M4ICB3DRD73UPANCNFSM4KZF3IWA .
There is a massive performance issue using readLipd in R caused by the argument ",guess_max = 1e6" in the call to read_csv in function read_csv_from_file.
Using the example file from http://nickmckay.github.io/LiPD-utilities/r/index.html, which is only 1.3 MB, uses over 7 GB of RAM.
Setting this to 1e3 fixes the problem.
I would make a pull request but I don't know enough about LiPD to know whether guessing from the first 1000 entries is likely to be good enough.