tmp data management

[eboileau]

Aims/objectives.

We need a general solution, e.g. files associated with a given session/user.

A clear and concise description of todo items.

• Data management (annotation).
• Data upload.
• Compare view. When a user makes a query, pybedtools is called, the question is how tmp files are handled on the server, are they all written to the same default tmp, no matter the user, and how to handle cleanup (bound to the python session)?

[eboileau]

Check https://daler.github.io/pybedtools/topical-design-principles.html#principle-1-temporary-files-are-created-and-deleted-automatically and https://daler.github.io/pybedtools/autodoc_source.html#pybedtools-setup-and-config

[HaraldWilhelmi]

Had a deeper look at pybedtools:

• The functions to handle temporary files are just thin wrapper around the standard tempfile module.
• The list of temporary files is held globally (pybedtools/filenames.py - TEMPFILES).
• The Bedtools objects don't have a __del__() or__exit__() method to clean up any temporary files.

Using pybedtools.helpers.set_tempdir to have per-request tempdirs will most likely not work - at least if we handle any requests asynchronously. So the most reasonable thing is a housekeeping cron job that does something like:

find /tmp -name 'pybedtools.*.tmp' -mtime +0.1 -exec rm -rf {} \;

[eboileau]

As discussed:

tmp upload management

• Compare view -> CompareStepB.vue, we use FileUpload with :maxFileSize="50000000". The API end point is transfer/tmp_upload (api/transfer.py). Saving of the file is handled by FileService.upload_default (services/file.py), and relies on Config.UPLOAD_PATH (which defaults to /uploads on the container).
• We need to keep the file path/name for display.
• A reasonable delay to keep files on the container would be 30 min.
• If we have to deal with error handling, there is already a skeleton for displaying messages in CompareView.vue, next to the submit button (C). You can try it e.g. by clicking the submit button w/o selecting one of the radio buttons. Errors arising from failed uploads (currently due to failed reading of the file, etc.) are also displayed there. See also #90

pybedtools tmp file management

Mostly happens when a user uses the Compare view, but also when creating dataset. In general, the simplest solution here is to let pybedtools do it's job, and we just add a "clean up" on top e.g. delete files that have not been accessed for more than 1 day.

For general info, see

• principle-1-temporary-files-are-created-and-deleted-automatically

• pybedtools.helpers.cleanup

• deleting-temp-files-specific-to-a-single-bedtool

[eboileau]

Latest changes to be tested on the server.