Cove-ocds: /tmp Disk Space management

[ghost]

Django is not clearing temporary files which has lead to disk space running out. See: ls -lh /tmp/*.upload*

Possible solutions:

• If the application / Django has any tools we can use to clean up these temporary files we can configure this to run daily.
• "tmpreaper" is a tool designed to clear down everything in /tmp
• When we next rebuild this server we can provision more disk space.

This issue may effect other Django applications so they need checking.

[jpmckinney]

Only this application receives a significant number of uploads, so I think it's limited to this and the cove-oc4ids server.

[jpmckinney]

I read https://docs.djangoproject.com/en/4.0/topics/http/file-uploads/#upload-handlers but not much there or on linked pages.

It seems like Django should be deleting the temporary files once they are closed: https://github.com/django/django/blob/main/django/core/files/temp.py

I'll check the CoVE code to see if it's leaving files unclosed (e.g. by leaking file handles).

[jpmckinney]

This comment suggests that the files need to be manually closed: https://stackoverflow.com/a/23657999

However, this comment explains that they are closed when the request is closed: https://stackoverflow.com/a/62357029:

• On non-Windows, Django uses tempfile.NamedTemporaryFile for file uploads https://github.com/django/django/blob/stable/3.2.x/django/core/files/temp.py
• NamedTemporaryFile.close() deletes the file https://docs.python.org/3/library/tempfile.html
• Request.close() calls NamedTemporaryFile.close() https://github.com/django/django/blob/stable/3.2.x/django/http/request.py
• Request.close() is registered at https://github.com/django/django/blob/stable/3.2.x/django/core/handlers/base.py
• HTTPResponseBase.close() calls registered methods (and swallows errors) https://github.com/django/django/blob/stable/3.2.x/django/http/response.py
• Per the code comment, the WSGI server must call HTTPResponseBase.close().

So a few possibilities:

• uWSGI isn't calling the close method during normal operations (unlikely)
• The request failed (e.g. upload too large) such that uWSGI didn't call the close method in those cases
• An error is preventing the files from being closed (e.g. a persistent reference to the file)

Anyway, that's as much time as I have. We can maybe use tmpreaper or a simple cron job.

FYI @Bjwebb