Closed torgeirl closed 5 months ago
Working on a stream-zip solution, have zip creation of feedbacksets for examiner done. Just need to implement it for the other places. It seems to work realy well. It does not have to use local storage since it does the zip creation in small chunks when the zip gets streamed to the user. So small memory footprint and no local storage needed.
Commited fix for this issue in 4b7965e
Changes to how zips are created makes CompressedArchiveMeta obsolete.
The use of Compressed archive metas was quite limited, but it was useful to easily find the path and (especially) its time of deletion without having to browse the contain folder(s). That is no longer displayed anywhere in Django admin?
We currently delete older ZIP files with automated scripts (ie python manage.py devilry_delete_compressed_archives --days 14
), but with S3 it's perhaps better to set a time to live upon upload?
When hosting Devilry as a service in a container orchestration environment, the memory allocation associated with the creation of compressed archives (up to multiple GBs) are problematic.
Suggested solutions a) either add an option for streaming the construction of the compressed archives using something like zipfly or stream-zip,
b) or implement an option where the ZIP creation use disk (in practice a persistent volume of the container's pod) instead of memory before it is moved from disk the storage location (as defined in settings; in our case S3 storage).