dandi / dandisets

749 Dandisets, 813.7 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets

10 stars 0 forks source link

minimize storage impact from logging #123

Open yarikoptic opened 2 years ago

yarikoptic commented 2 years ago

122 enabled logging into a file at DEBUG level, which could come of great use for troubleshooting in case of errors.

But each log file is about 19MB and in a day we already amassed 0.5GB of them. Clearly it is a bit "wasteful" and we should do something about that. Some ideas

keep the log only if run errored out and otherwise remove (should probably be possible to setup I bet)
gzipping folds it to ~10% of its size. May be we should store them gzipped anyways -- why to waste space (although IIRC this partition also has file system level compression enabled)

@jwodder -- wdyt?

jwodder commented 2 years ago

@yarikoptic Deleting the logfile on successful exit seems easier to implement.