Removing old data from public ETF instance

[michellutz]

We are having issues again with the storage space available in our public ETF instance (see this recent comment).

We have already a regular cron job running that removes all the files older than 15 days inside the folder .etf/testdata.

However, doing some more digging in the other data folders described in 3.3. ETF data directory structure of the Admin manual, we found that the following ones still contain vast amounts of data:

• .etf/ds/db/data (43GB)
• .etf/ds/attachments (52GB)

According to the documentation,

• the ds/db/data/ folder contains "other databases that are created during test runs to speed up test queries", and
• the ds/attachments contains "files that are downloaded or created during a Test Run".

I assume that, if we want to keep only the test reports for the last x (say 15) days available in the application, we can also remove any sub-directories and attachments in these folders that are older than 15 days. Correct?

Furthermore, I am wondering if we could "clean up" data also in the other folders, e.g.

• the obj folder containing XML representations of all framework items (Executable Test Suites, Reports, etc.), or
• the db/data/etf-ds directory containing the database with these items

[carlospzurita]

• The folder ds/attachments can be safely cleand up, as it stores the files tested (e.g. the metadata .xml).
• In the folder ds/obj, the only files that can be removed without affecting the ETSs are those that are named TestRun-.... These are the files that contain the test reports in .xml format.
• The directory db/data/etf-ds should not be removed. Other folders in db/data apart from this can be deleted because they are auxiliary databases.

Anyhow, the ETF is configured by default to execute a job every midnight, checking the date of the TestRun files and deleting those that are older than the number especified in the etf/config/etf-config.properties file.

[michellutz]

@df-git Could you pls have a look at the config files and the logs to see if the TestRun files are deleted as they should or not.

[carlospzurita]

If you can provide us with the log files, and the ETF folders that you have in your environment, we can also take a look at it.

[carlospzurita]

Looking at the ETF logs that were provided, it seems like the scheduled task for cleaning test objects and results is working every night, and reports that a number of files were cleaned. There are a couple of TestResult that are reported as non-existant:

• d7416ecf-3dd7-4291-88df-aa8ee47e8ae6
• 3a327d1f-2add-46b4-880b-617a3df02b53

Can you check if those files are still in the etf folders? Also, can you rebuild and redeploy the application, but changing the properties

• etf.testobject.temporary.lifetime.expiration
• etf.testreports.lifetime.expiration

to 480?This will put an expiration date of 8 hours, ensuring that everything gets cleaned at midnight. This way you can monitor everyday what are the files that are not deleted.

In the meantime we are checking in our premises the behaviour, and we will get back to you as soon as we have more information.

[carlospzurita]

We checked on our premises, and it seems that the even thought the references are removed from the BaseX storage, the files asociated with the test runs in ds/attachements are not removed. Did you executed the cleanup on your deplyment? how is it now?

[jonherrmann]

Closing this in favour of:

• issue https://github.com/etf-validator/etf-webapp/issues/208
• could be fixed by https://github.com/etf-validator/governance/issues/90