Closed anishthite closed 3 years ago
Thanks for writing down this suggestion! @leogao2, you’ve had the most experience processing these data sets. What do you think is a good size chunk that someone can use to ensure the data is being downloaded and processed correctly?
Most of these datasets are provided as big tarballs so we'd have to host the small ones separately which would add a lot of complexity.
Hmmm. I hasn’t thought about that. And unzipping a tarball is an all-or-nothing thing right?
You can partially unzip a tarball by separating it by file (https://unix.stackexchange.com/questions/42198/untar-only-a-certain-number-of-files-from-a-large-tarball/42199).
I think that adding that functionality would be highly useful then.
@leogao2 @researcher2 I believe the branch fix_size is the implementation of this functionality. Is that correct? If so, is it finished? What more needs to be done?
That branch was originally to do some changes to the output of size() on some datasets and is unrelated to this, and then I used it for testing on Hetzner. I will delete it from the repository to avoid further confusion.
That branch was originally to do some changes to the output of size() on some datasets and is unrelated to this, and then I used it for testing on Hetzner. I will delete it from the repository to avoid further confusion.
Very well. How about this issue? Has a fix been merged or is it still outstanding?
to assist with data exploration and testing (both ours and that of other people) we should add a “small” flag that causes it to download a small amount of data (size TBD... 10M per data source? --Stella