mountetna / magma

Data server with friendly data loaders
GNU General Public License v2.0
5 stars 2 forks source link

Da tsv large tables #68

Closed dabrau closed 6 years ago

dabrau commented 6 years ago
graft commented 6 years ago

I think tsv writer should be a class rather than a module (instantiated by e.g. Retrieve), I find it clearer to see where methods are coming from that way.

graft commented 6 years ago

I would favor getting rid of the gzip stuff, since it's easily done with gzip -c in a shell script and is probably faster that way; in fact I'd get rid of all file writing entirely and just write to stdout. It is the most unix-like, the most flexible, and makes the least assumptions about how this command is used (for example, you can feed it to grep that way). You can still fake this by passing in /dev/stdout as the command parameter but I think it works better without a file argument. Probably means you can get rid of your ModelUnloader class.

graft commented 6 years ago

Other than that shmeeny spec change this looks good to me.