Open slazicoicr opened 6 years ago
Thanks for reporting.
options(warn=2)
was anticipated and does not generate this warning. But trapping warning()
via tryCatch()
wasn't anticipated unfortunately. Something 'unknown' was anticipated, though, and that's the coping mechanism you're seeing: it cleans itself up upon next call and issues a good warning. It could have been a lot worse.
I don't know of a way to know at R level whether tryCatch(..., warning=)
has been specified as something that halts, or not. Any attempt at that is likely to messy and fragile.
One solution might be for DTWARN
in fread.c
to cache the warning(s) in a private buffer and then call R/Python's warning() on exit after freadCleanup()
.
what if we surround the entire R fread(...)
call into one big tryCatch(..., finally=freadCleanup)
call?
That's a neat idea!
Would be adding tryCatch(..., finally=.Call(CfreadCleanup))
around the .Call(CfreadR)
here I guess : https://github.com/Rdatatable/data.table/blob/master/R/fread.R#L101
Hello. I'm running into this same problem when trying to catch the "parsing-problems-related" warning messages in a custom file-reading function I'm creating. I've seen other packages using readr::problems
to do this, but I'm trying to stick to data.table
on this.
Do you have any suggestions on how to do this while staying away from this warning? I'll post the issue I created on my own repo below to exemplify the error I'm facing.
You can see that this line: gtfs <- read_gtfs("../msc-thesis/otp/graphs/rio/ola.zip")
is what causes the warning to be thrown in the first place. When I try to use the same command again, it throws an unzip
-related error. However, if I specify the second argument as stop_times
(which means I'm only unzipping this file and overwriting the others [such as stops
, that causes the problem]) I can subsequently run gtfs <- read_gtfs("../msc-thesis/otp/graphs/rio/ola.zip")
, apparently because of the cleanup.
I actually have an on.exit()
call to unlink the temporary directory gtfsdir
created, but when I have parsing failures (i.e. when I catch the warnings) the directory is not removed, even with force = TRUE
. I have also tried to manually remove the directory, but I get the message that the file is being used by another program thus the directory cannot be removed.
ps: "não foi possível abrir o arquivo" means "could not open the file"
Also, another problem is that fread
seems not to do well with tryCatch
. Check this:
gtfs <- read_gtfs("../msc-thesis/otp/graphs/rio/ola.zip")
#> Warning message:
#> In read_gtfs("../msc-thesis/otp/graphs/rio/ola.zip") :
#> Parsing failures while reading the following file(s): trips, stops
gtfs <- read_gtfs("../msc-thesis/otp/graphs/rio/ola.zip")
#> Error in utils::unzip(path, files = files_to_read, exdir = temp_dir, overwrite = TRUE) :
#> não foi possível abrir o arquivo 'C:/Users/Usuario/AppData/Local/Temp/RtmpqS6Xww/gtfsdir/stops.txt': Invalid argument
gtfs <- read_gtfs("../msc-thesis/otp/graphs/rio/ola.zip", "stop_times")
gtfs <- read_gtfs("../msc-thesis/otp/graphs/rio/ola.zip")
#> Warning message:
#> In read_gtfs("../msc-thesis/otp/graphs/rio/ola.zip") :
#> Parsing failures while reading the following file(s): trips, stops
I actually suppress a possible warning when reading the gtfs due to how the function is structured. If I remove these warning supression you can see that fread
does some cleaning after the "invalid argument" error.
gtfs <- read_gtfs("../msc-thesis/otp/graphs/rio/ola.zip")
#> Error in utils::unzip(path, files = files_to_read, exdir = temp_dir, overwrite = TRUE) :
#> não foi possível abrir o arquivo 'C:/Users/Usuario/AppData/Local/Temp/RtmpqS6Xww/gtfsdir/stops.txt': Invalid argument
gtfs <- read_gtfs("../msc-thesis/otp/graphs/rio/ola.zip", "stop_times")
#> Warning message:
#> In data.table::fread(file.path(temp_dir, file), nrows = 1) :
#> Previous fread() session was not cleaned up properly. Cleaned up ok at the beginning of this fread() call.
Related data.table issue: https://github.com/Rdatatable/data.table/issues/2904
I have also encountered this problem recently. Do you have any good solutions? I need to read multiple CSV files in turn. Some files need to be separated by other symbols according to the warning information instead of using the default double quotation marks
Thank you
Hi @zoushucai, I started using withCallingHandlers()
instead of tryCatch()
, because it doesn't interrupt the running process. Perhaps it could be well suited for your needs as well.
Thank you for your suggestion. My problem has been solved @dhersz
That's a neat idea! Would be adding
tryCatch(..., finally=.Call(CfreadCleanup))
around the.Call(CfreadR)
here I guess : https://github.com/Rdatatable/data.table/blob/master/R/fread.R#L101
Hi @mattdowle, could you please kindly point out how should I use
...
finally=freadCleanup
...
or
...
finally=.Call(CfreadCleanup)
...
Because every time I try above I'll get a 'C symbol name "freadCleanup" not in load table' error. Thank you in advance!
The three lines below work as expected:
The first and third line read just fine and the second line throws a warning.
The issue happens when the line that throws the warning is wrapped up in a tryCatch block
Calling the last fread throws a warning message, even though it should work just fine:
sessionInfo: