Create portable Docker image

[bugzilla-to-github]

This issue was created automatically by a script.

Bug 1513104

Bug Reporter: dane.molotok@gmail.com CC: @adngdb, @Pike, @eemeli, erickwilder@gmail.com, keunes@mailbox.org, @mathjazz, me@avm99963.com, mlzc@hey.com, santosh.prajapati@ibriz.ai

Pontoon is heavily tailored to be deployed on Heroku. With some massaging, the work done for the dev Docker container could be used to create a production-ready Docker image. This could be published on DockerHub and used by folks who want to run Pontoon in their existing infrastructure.

[bugzilla-to-github]

Comment Author: dane.molotok@gmail.com

I have this working on our Mesos cluster. I will pull request shortly.

[bugzilla-to-github]

Comment Author: @Pike

Created attachment 9085138 GitHub Pull Request [Closed]

Attached file: file_1513104.txt (text/x-github-pull-request, 44 bytes) Description: GitHub Pull Request [Closed]

[bugzilla-to-github]

Comment Author: erickwilder@gmail.com

The linked pull request was closed and I was wondering if there is any intention to resume the work on this topic. Do you need any help on that? I was planning to deploy Pontoon on a Kubernetes cluster but if we have an official docker image this would be much more beneficial to the whole community, not only a few people trying the same as I am.

[bugzilla-to-github]

Comment Author: @mathjazz

Hey erickwilder, we'd be more than happy to mentor you!

I'm leaving NI for :adrian (on PTO until Thursday), who's been actively involved in the work that has been done so far. He should be able to give you more concrete pointers.

[bugzilla-to-github]

Comment Author: @adngdb

Hey Erick,

We would be very glad to have your help on this. I tried to write our Pontoon Dockerfile so that it is as production-ready as possible. There are, however, a few things that need to be changed in order to make the current one actually fit for a production deployment.

What would be really great, if you could manage that, would be to make our main Dockerfile production-ready, while still being able to use it for local development with the docker-compose file on top of it. That probably means making it a multi-stage build, with the dev image having all tools used for building (as we need them regularly for development) and the prod one having as little as possible.

Anyway, please do feel free to work on this, and I'll be happy to help!

[bugzilla-to-github]

Comment Author: @Pike

As this has come up again, IMHO it's a lot more important to create a production-ready version of https://github.com/mozilla/pontoon/blob/master/docker/run_webapp.sh, akin to https://github.com/mozilla/pontoon/blob/master/Procfile#L1.

It's also an interesting question if the daemon should use the same image, or a different one?

And, in the context of something like kubernetes, where should ./manage.py migrate run?

[bugzilla-to-github]

Comment Author: Michael Lin <mlzc@hey.com>

(In reply to Axel Hecht from comment #6)

As this has come up again, IMHO it's a lot more important to create a production-ready version of https://github.com/mozilla/pontoon/blob/master/docker/run_webapp.sh, akin to https://github.com/mozilla/pontoon/blob/master/Procfile#L1.

It's also an interesting question if the daemon should use the same image, or a different one?

And, in the context of something like kubernetes, where should ./manage.py migrate run?

Coming from https://github.com/mozilla/pontoon/pull/1779.

We are running a production instance at https://translate.skills.network on K8S. It's designed in a way that no change is made to Pontoon source code and we can still run it in a really flexible way.

We take the Dockerfile from the Pontoon repo without modification and publish it to Docker Hub. Here's the process of building and pushing the image. https://github.com/ibm-skills-network/pontoon/blob/master/.github/workflows/push-image.yml I don't see why the current Dockerfile is not suitable for production deployment if we ignore the fact there are security vulnerabilities in packages and there are a bunch of dev dependencies that are not needed :). The point is, it has everything we need to get it working.

In the Helm Chart (a package manager for K8S), we try to replicate how Pontoon is run on Heroku (The Procfile). https://github.com/ibm-skills-network/charts/tree/master/charts/pontoon

1. pontoon-deployment, the web process. NOTE, we don't use the default entrypoint but use gunicorn to run the production server.
2. ponton-worker-deployment, the worker process.
3. ponton-migration-job. It will be run whenever the chart is installed or updated.
4. ponton-sync-projects-cronjob. cronjob that runs the sync_projects command periodically.

It also comes with some features that we think are helpful

• mount multiple SSH keys into the container
• ability override the default Django settings
• opt-out strict SSL, good for running a test instance.

[bugzilla-to-github]

Comment Author: santosh.prajapati@ibriz.ai

Hi Erick,

1. Do you have any documents or steps to deploy in other cloud providers like AWS?
2. What are the things that must be considered if we deploy pontoon in the production with docker-compose?

I really appreciate any help you can provide.

[bugzilla-to-github]

Comment Author: keunes@mailbox.org

Just to note that there's I too have an interest in this. I'm wondering what's necessary to get this done. There's interest in the result, there's people who are willing to work on it, others using a docker image in production already, and Mozilla seems to have indicated willingness to mentor those getting this ready.

Is there any way I can help to move this forward? I could help organise a quick call, for example (I'm not a developer and don't know much about docker).

[bugzilla-to-github]

Comment Author: @mathjazz

As mentioned on Discourse, this is one of the items that Mozilla doesn't directly benefit from, because it doesn't use it. Hence the P5.

We need someone to pull this forward, provide the patch and we'd be happy to provide feedback and review.

You are welcome to join us at the weekly call every Thursday at 1700 CET at https://mozilla.zoom.us/j/208705274, where we can discuss the next steps.

[bugzilla-to-github]

Comment Author: @eemeli

Mentioning here in case there are any interested parties: Some Pontoon Dockerfile progress is starting to happen with https://github.com/mozilla/pontoon/pull/1996, which might get followed by further changes leading to an improved experience for those looking to use Pontoon via a portable/prod-ready Docker image.

To be clear, this is a part of an effort to improve the developer experience and e.g. reduce unnecessary Docker image re-builds, rather than an effort to explicitly produce a prod-ready Docker image.

[Freebios]

I had a use-case for this, I'm working on a PR for relatively soon.

I'm very new on the project and I might miss important things but from what I saw, the static files building pipeline has to be changed, right now there is too much stuff built into a mix of python and node, leading to huge 2.6GB final image, which is even for dev purpose, very big.

I'll try to get stuff done for this week!

[keunes]

Hi @Freebios Without any expectations: For the open source project AntennaPod I'd like to use Pontoon for our website translation (could use Weblate, but as a translator I have the impression Pontoon is a bit easier to use). The project maintainer, since it's my request, asked if I could look into setting up a translation system. I just had a look at your fork and it seems there's some activity in the dockerize branche, am I correct? I'd be happy to share experiences, if it's of any help :-) (but I have no experience with docker or Pontoon... yet)

[Freebios]

Sure, feel free to DM me on Twitter (my handle is on my profile). Currently, I have a working docker image deployed on kube without much issue. If you want I can give you a hand on deploying it on your environment :+1:

[keunes]

Thanks @Freebios, I'll send you a DM on Twitter. Would like to get my hands dirty with Pontoon and get some experience with it. But I just saw @mathjazz's comment on your PR #2320, so that outlook makes it unlikely we'll use Pontoon in production, for now, for AntennaPod.

@mathjazz, you mention a production-ready Docker should wait for the (frontend) build process refactoring. Would there be any rough estimation of when that work might be picked up? Just to have an impression of whether we're talking weeks, months, or more direction of 1(+) year.

[eemeli]

Would there be any rough estimation of when that work might be picked up? Just to have an impression of whether we're talking weeks, months, or more direction of 1(+) year.

Of those options, "months" is the likeliest choice.