Improve the Celery setup on the development environment.

[bugzilla-to-github]

This issue was created automatically by a script.

Bug 1612580

Bug Reporter: @jotes CC: @adngdb, @Pike

The configuration of Pontoon's development environment doesn't include all of the components that are required by the Celery instance on Prod. It makes issues like e.g. #1477570 harder to diagnose because e.g. you don't see if Celery is fetching tasks correctly.

I think it would be valuable addition to our repository.

Pros:

• development environment is closer to the state on Prod Cons:
• Containers for RabbitMQ and Celery will consume additional resources

Additionaly, I would still run tests with CELERY_ALWAYS_EAGER=True because, without that, it will make testing a little bit harder and would require additional amount of work.

I've implemented a PoC of this setup here: https://github.com/mozilla/pontoon/compare/master...jotes:heroku-test-sync-ampq

Basically, I added all required components (RabbitMQ and a Celery worker) in a separate Dockerfile.

:Pike, :adrian What do you think about this?

[bugzilla-to-github]

Comment Author: @Pike

I guess it's OK to have an optional set of services, and, TIL, adding another compose config is the right way to do that.

I'd stick to a single docker file for the webapp and worker, as that's what we're doing in production. Also, a single config for both.

If you're willing to, would you drive-by-fix https://github.com/mozilla/pontoon/blob/76e6110196d05e4f7209b445abe6f7d604a0e837/Makefile#L37 to not create a backup file with the -e as extension? Replace the cp and sed -i with a sed > foo ?

[bugzilla-to-github]

Comment Author: @jotes

:Pike Sure, I'll replace it.

[bugzilla-to-github]

Comment Author: @adngdb

I'm absolutely in favor of doing this. I agree with Axel though that it would be better with a single "Pontoon" image, which runs different scripts. Not sure if that would work, but I would try creating a run_sync.sh or run_celery.sh script (similar to run_webapp.sh) and simply set that as the entry point for the celery/sync container.

Side note on naming: I like to name things for what they do rather than for what they are built with, so I wouldn't use the name "celery" here, but call it "sync" instead.