search

cboettig / neonstore

:package: A local content-based storage system for NEON data
https://cboettig.github.io/neonstore
Other
8 stars 5 forks source link

Adding the check for gunzipped files in the unzip_all function. If t… #13

Closed rqthomas closed 4 years ago

rqthomas commented 4 years ago

Adding the check for gunzipped files in the unzip_all function. If they are present it uses R.utils::gunzip to uncompress them. This adds a new package requirement (R.utils) and I think that R.utils::gunzip returns information that we don't want printed to the screen. Maybe there is a better gunzip function.

rqthomas commented 4 years ago

The R.utils package need to be added to the documentation so that it imports it.

cboettig commented 4 years ago

Yeah, NEON shouldn't really be gzipping hd5 files, since hd5 standard supports gzip internally already and they should use that. (hd5 is already a binary file, unlike csv). In that way, the data also stays compressed on disk and thus takes up less space, and is just uncompressed when it is read in by any hd5 library. I've filed an issue with NEON about this already, and hopefully they will stop gzipping hd5 like this in the future. It's doubly annoying, because it isn't necessary for text files either -- gzip is a streaming compression, and base R already defines gzcon which can unzip a compressed connection on read-in.

But since they are doing so now, we probably do need this at least for the hd5 files. Looks like CI checks aren't happy; we'll just need to declare the imported function and package dependency. (add @importFrom R.utils gunzip and run usethis::use_package("R.utils") to add the additional dependency)