Support object store besides file store for persistancy in Weblate

[rhofer]

In order to separate application layer from persistancy layer in Openshift, the persistant volume clame in Openshift should not be used (files store). Instead, an object store as AWS S3 should be used, as django-storages would allow.

Question: is Weblate already supporting this today?

[nijel]

The problem is that we don't only need generic storage layer as django-storages is, but we rely on filesystem on which we can store working copies of the VCS repositories and files to execute external tools (eg. git, gettext). What could work here is focusing on Git only and storing the Git objects in a database and perform the operations on that or using temporary working copies. This not something we can do in forseable future. The other part is fulltext index, which will be gone in the 4.0 release, see https://github.com/WeblateOrg/weblate/issues/2825.

For user uploaded content, it might be possible to use django-storages. It would certainly work for screenshots, but with fonts it would be again tricky, because the the rendering backend supports only loading fonts from a local directory. There seems to be a solution in C to load fonts directly, but that doesn't work out of box in Python as it touches too low level parts which are not exposed in the Python bindings.

TLDR: It's probably not impossible, but would need major rewrite and it's certainly not on roadmap for now.

[nijel]

Summary of what we store in the DATA_DIR and possible solutions:

• avatar-cache - this does not need to persist, can be rebuilt, not all workers need it
• backups - needed on the backup worker only for doing backups
• cache - fontconfig local cache, this really should live inside /app/cache in the Docker, will be fixed in https://github.com/WeblateOrg/weblate/pull/8636
• celery - celery beats schedule, should be addressed by https://github.com/WeblateOrg/weblate/pull/8635
• fonts - fonts used by fontconfig/pango/cairo, these need to be on the actual filesystem and kept in sync
• home, ssh - configuration for command line tools invoked by Weblate, some files can be regenerated, but some need to persist (SSH keys, borg backup keys)
• media - user uploaded screenshots, this one should be ready for using other storage
• projectbackups - these should be possible to rewrite using Django storage layer
• static - is already using /app/cache in Docker
• vcs - version control data, right now Weblate supports Git, Mercurial and git-svn, so solution for both is needed, for Git https://github.com/bgahagan/git-remote-s3 might be a solution

Other affected features of storage change:

• Backups rely on having all repos in DATA_DIR, so doing full backup will need full repo sync as well