"Shard" times.csv.bz2 into a one-to-many file structure.

JamesSaxon / access-old

Python module to calculate spatial access scores.

http://access.readthedocs.io

BSD 3-Clause "New" or "Revised" License

4 stars 0 forks source link

"Shard" times.csv.bz2 into a one-to-many file structure. #10

Closed JamesSaxon closed 5 years ago

JamesSaxon commented 5 years ago

[x] Create a single bash script for this.
[x] Input is the bzipped times files. Output is a directory of the form state/county/tract.csv, e.g., for me 17/031/410900.csv.
[x] File format is going to be JUST dest,cost. Destination is the is the 10-digit geoid string (or, 9 or 10 digit number) and the float. The origin is implicit in the file name.

yatlas commented 5 years ago

My script works, however it takes a while. Part of it has to do with the size of the file, but I'm not sure if mine takes longer than expected. I added it to meta branch.