DB history table housekeeping option

cylc / cylc-flow

Cylc: a workflow engine for cycling systems.

https://cylc.github.io

GNU General Public License v3.0

335 stars 94 forks source link

DB history table housekeeping option #4608

Open hjoliver opened 2 years ago

hjoliver commented 2 years ago

Currently, the task_events DB table grows without bound.

This is desirable from a provenance perspective, for finite-length science experiments; but probably not desirable for large endlessly-cycling weather prediction workflows (for example).

We should keep all information by default, but should we allow old info to be deleted at some cut-off point behind the active window? (By number of cycle points and/or by cycle point offset and/or by real time offset interval).

oliver-sanders commented 2 years ago

Related tickets:

https://github.com/cylc/cylc-flow/issues/3872 (would reduce the size of the DB making housekeeping less necessary).
https://github.com/cylc/cylc-flow/issues/4609 (would require housekeeping to be turned off).