In erronuous situations it may happen that rows in the database get left. There is currently no mechanism that cleans them up. A sync or async thread (up to the developer) in the orchestrator should loop through the database checking if a row is older than 48 hours. In case the row is older than 48 hours, the row may be deleted outright.
This component should safe (in memory) the datetime when the component was initialized. This time may be used to check how long the orchestrator has been active for. Only after the orchestrator has been online for the time period for which is checked (48 hours) should the orchestrator be confident the job/row is actually 48 hours without successful completion. Otherwise, the orchestrator may have been offline for 48 hours or longer and the rows would be removed erronuously.
The component should receive the PostgresInterface as a constructor argument. The component MUST be isolated from other components in the orchestrator.
Please make sure to put any constants in logical places and make the 48 hours timeframe configurable. We use .env files as configuration files so please extend .env-template with any new configuration parameters.
Where to add to orchestrator:
main.py:main should initialize this component (with the other components) (and should not yet START the component!)
main.py:Orchestrator.init should expect this component as a constructor argument.
main.py:Orchestrator.start should start this component.
main.py:Orchestrator.stop should stop this component.
Relevant functions:
postgres_interface.py:PostgresInterface.get_all_jobs may be used to retrieve all jobs.
postgres_interface.py:PostgresInterface.delete_job may be used to delete a single job.
db_models.job.py:JobDB contains a number of relevant fields: running_at, submitted_at, registered_at
The following logic may be used to determine if a job too old:
If the orchestrator has been running for 48 hours or longer AND:
If JobDB.running_at exists and is older than 48 hours the job is too old.
OR If JobDB.running_at is None and JobDB.submitted_at exists and is older than 48 hours the job is too old.
OR If JobDB.running_at is None and JobDB.submitted_at is None and JobDB.registered_at is older than 48 hours, the job is too old.
At least the function to determine if a job is too old must be unit tested. Other functions may be unit tested when possible or effective.
In erronuous situations it may happen that rows in the database get left. There is currently no mechanism that cleans them up. A sync or async thread (up to the developer) in the orchestrator should loop through the database checking if a row is older than 48 hours. In case the row is older than 48 hours, the row may be deleted outright.
This component should safe (in memory) the datetime when the component was initialized. This time may be used to check how long the orchestrator has been active for. Only after the orchestrator has been online for the time period for which is checked (48 hours) should the orchestrator be confident the job/row is actually 48 hours without successful completion. Otherwise, the orchestrator may have been offline for 48 hours or longer and the rows would be removed erronuously.
The component should receive the PostgresInterface as a constructor argument. The component MUST be isolated from other components in the orchestrator.
Please make sure to put any constants in logical places and make the 48 hours timeframe configurable. We use .env files as configuration files so please extend .env-template with any new configuration parameters.
Where to add to orchestrator:
Relevant functions:
running_at,submitted_at,registered_atThe following logic may be used to determine if a job too old:
JobDB.running_atexists and is older than 48 hours the job is too old.JobDB.running_atis None andJobDB.submitted_atexists and is older than 48 hours the job is too old.JobDB.running_atis None andJobDB.submitted_atis None andJobDB.registered_atis older than 48 hours, the job is too old.At least the function to determine if a job is too old must be unit tested. Other functions may be unit tested when possible or effective.