WeblateOrg / weblate

Web based localization tool with tight version control integration.
https://weblate.org/
GNU General Public License v3.0
4.62k stars 1.02k forks source link

Add Heroku support #732

Open nijel opened 9 years ago

nijel commented 9 years ago

It should not be hard to implement automated deployment to Heroku, somebody has even started that: https://github.com/blockgiven/weblate/commit/e155371a932a09c4ffca91d798a5d232866b7b75

ravishi commented 9 years ago

I've just tried running it on Heroku, but found it impossible to do so because weblate expects a persistent filesystem (data/vcs/{project} for instance), and Heroku doesn't provide that. It runs, but every time you deploy a new version the data folder is erased.

nijel commented 9 years ago

Isn't there option for persistent file storage? Indeed Weblate will not work without it (the data/ directory is used for storing repositories, fulltext search index and some other stuff).

ravishi commented 9 years ago

Yeah, have just figured that out. I'm sorry for the noise.

nijel commented 9 years ago

Can you please share something from your testing? Either as documentation or as code?

ravishi commented 9 years ago

Of course! I'm currently running it on Dokku[1], which is pretty much like Heroku (it uses Heroku buildpacks, at least), so some of the findings can be useful.

There is a branch at labhackercd/weblate/tree/dokku.

The installation is pretty much like running any other Django project on Heroku, with only three issues so far:

I'll give our setup a little more time to detect any other issues, since I'm not that used to Heroku yet. If anything pops up I'll report here.

[1] http://progrium.viewdocs.io/dokku/

acamara7es commented 8 years ago

We have tried to run Weblate in Heroku too, but have the same problem (need persistent filesystem). We had thinking about it and we found some ideas for this but, after read the previous messages in which nijel says that the data folder is used for fulltext searches too, only one has (maybe) a bit of sense.

In fact this idea comes from a workmate that knows django better than me so I'll investigate it to give more information about it if anyone think that this has sense. How I said, I'm not an expert in django and I haven't time enough to become me an "expert" of django and then understand how is weblate internally (at least in a short period of time).

I hope this helps. We actually are using oneskyapp to manage our translations but for me Weblate is better and I want to use Weblate again (just needs to work on Heroku...)

(One of the older ideas was try to liberate weblate of the internal repositories but it no longer makes sense because now I know that the "data" folder has more functions.)

nijel commented 8 years ago

I'm missing a bit what needs to be changed. The major thing data is used for is storing VCS repositories, what is pretty much essential feature on which Weblate is built (there is currently no other way to get translations out of it), so this really has to be filesystem. There is fulltext data as well, but that is not that important (and we need to look for another implementation anyway, see #800).

Alir3z4 commented 5 years ago

Having dependencies on the local file system makes it difficult to:

Since data_dir is required for VCS files, then there are 2 possible solution to fix it.

  1. Use FUSE mount to attach external/cloud storages such as s3 or ceph, etc:
  2. Compress the VCS directory into a single file and upload to S3 via django filefield model backed by django-storages.

How option 1 would work

You need to build the S3 Fuse, by a custom buildack and connect the bucket to your app as DATA_DIR. Heroku allows pushing docker images as well, so if you don't want to do buildpacks, you can build the docker image with S3 Fuse support and deploy to Heroku.

How option 2 would work:

Every time the VCS changes, compress and save it to S3 if there's a high load, it can be done each hour or each day, etc.

Whenever app starts, if data_dir is empty the compressed VCS files get downloaded and extracted into data_dir.

However, such solution will not be that reliable because still a real file system needs to be present and and always uploading and downloading the VCS from external storage would be a pain and hacky.

Should keep in mind, that heroku apps also have a size limit and having such stuff on heroku can exceed the storage space as well.

Overall, not a good idea.

comradekingu commented 4 years ago

@Alir3z4 For https://devcenter.heroku.com/changelog-items/1145 or otherwise? Is it a better idea to use https://www.heroku.com/postgres in a different deployment or not? Not every use-case is the heaviest one.

Alir3z4 commented 4 years ago

@comradekingu slug size is not the main issue, the fact that on each deployment the heroku dyno/instance will be completely wiped out, that's why using FUSE to S3 will be better option here, since the data are no longer on the dyno file system but kept on s3 and regardless of how many deployments/restarted/crash is applied, the files will be kept untouched.

patcon commented 4 years ago

fwiw I've had some success getting an alternative open source "continuous localization" project called https://mojito.global working on Heroku: https://github.com/patcon/polis-translations

It's developed by a coder on staff at Pinterest (formerly Box) and stores all data in MySQL (though other datastores should be possible)

tvb commented 4 years ago

I would really like to see Weblate become available with Heroku. If it would be possible to move the persistent storage to postgres I think it would be a great solution.

nijel commented 4 years ago

I don't think PostgreSQL is solution for all our needs - fonts and images is not really the kind of content you would like to store in the PostgreSQL. The same probably applies for Git repositories.

tvb commented 4 years ago

Well, fonts and images could be easily hosted on a CDN/Object Store right? Git is something else, hmm.

nijel commented 4 years ago

Yes, I'm just saying that PostgreSQL will not address everything and this is not a simple task.

mpachas commented 1 year ago

S3 should work as a repository filesystem . All other persistent files could be managed by s3, but you should rewrite everything related with file access (to be pluggable) with s3boto storages or simmilar.

nijel commented 1 year ago

S3 remote might address sharing the Git repositories data, thanks for sharing the link.

you should rewrite everything related with file access (to be pluggable) with s3boto storages or simmilar

The most problematic will be probably pango/cairo/fontconfig stack used for rendering images with text. I really don't see an alternative here, so the solution might be to sync this from S3 before calling these.

nijel commented 1 year ago

Let's keep this issue focused on the actual Heroku integration – there is another issue for the storage: https://github.com/WeblateOrg/weblate/issues/2984. I've just posted there an up-to-date summary.