socrata / datasync

Desktop / Console application for updating Socrata datasets automatically.
http://socrata.github.io/datasync/
MIT License
80 stars 33 forks source link

Allow the source CSV to be compressed #64

Open levyj opened 10 years ago

levyj commented 10 years ago

If DataSync-FTP already compresses the file before transferring, which I think is the case, allow the user to indicate that the file is already compressed in whatever format you use. This creates options for the user if it might happen to be to the user's advantage -- such as if the file has to be transferred internally as part of the ETL process or if the partition holding the file has limited space. It also might make some archiving processes simpler by allowing the compression to take place at the time of file creation instead of having to do it after the DataSync job.

These situations (or others that are not occurring to me) may or may not come up often but if you are going to compress the file, anyway, I am hoping that skipping that step is easy and gives the user more options.