Automated gz/bz2 decompression for RawAnalyzers

mami-project / pto-postgres

PostgreSQL backend for Path Transparency Observatory and associated utilities

0 stars 1 forks source link

Automated gz/bz2 decompression for RawAnalyzers #14

Open britram opened 7 years ago

britram commented 7 years ago

When reading from a raw file which is gz or bz2 compressed, the analysis runtime should automatically provide a decompressed stream to the rawanalyzer

FMNSSun commented 7 years ago

ptodev=> SELECT DISTINCT (metadata -> 'format')::varchar FROM uploads;
          varchar           
----------------------------
 "fjson-bz2"
 "ps-ecn-fjson-bz2"
 "csv-bz2"
 "ecnspider1-zip-csv-ipfix"
 "ipfix"
 "json"
(6 rows)

Current file formats of existing uploads.