delete sequences WHILE aggregating

renepickhardt / generalized-language-modeling-toolkit

Generalized Language Modeling toolkit

http://glm.rene-pickhardt.de

51 stars 17 forks source link

delete sequences WHILE aggregating #33

Closed renepickhardt closed 10 years ago

renepickhardt commented 10 years ago

the aggregator task shoul delete the sequence file after aggregation

This is absolute necessary in order to free disc space.

This can be achieved by changing the internal API from iostreams to files. Our logging system would also benefit from this mechanism since the aggregatorTask could output which file it is currently aggregating. This would help if there are any errors to be handled and also to see some progress

lschmelzeisen commented 10 years ago

Implemented with commit f28db891ccdc90f32cf01712878822c83c552a65. You just need to set deleteTempFiles = true in config.txt.