Processing workflow (compression)

Kaszanas / SC2InfoExtractorGo

GNU General Public License v3.0

0 stars 2 forks source link

Processing workflow (compression) #1

Closed Kaszanas closed 3 years ago

Kaszanas commented 3 years ago

Currently unpacking replays is resulting in x40 data multiplier. Bzip2 after tests was the best compression method for sample of produced .json files.

Workflow should be as follows:

Convert 10 000 replays to JSON
Compress with bzip2 to another output directory "processed_0.zip"
Delete files that were packed
Repeat

Kaszanas commented 3 years ago

This is implemented in the current code.