jgmize/mozorg-env - Githubissues

Mozilla.org development environment
Rationale
- www.mozilla.org is in transition from the following technologies
- Apache
- PHP
- Riverbed
- MySQL
- The majority of the pages on the site are now powered by [[https://bedrock.readthedocs.org][Bedrock]], but
- The traffic goes through a recently renamed Riverbed product formerly known as the Zeus Load Balancers (ZLB)
- Most requests are served by the riverbed cache
- All non-cached requests go to Apache
- Apache is configured to route most traffic through [[http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/includes/prefetch.php?view%3Dmarkup][prefetch.php]] via an [[http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/.htaccess?view%3Dmarkup][.htaccess]] file in the root of the mozilla.com svn repo
- In order to route pages in apache to bedrock, a passthrough RewriteRule to needs to be added to [[https://github.com/mozilla/bedrock/blob/master/etc/httpd/global.conf][global.conf]]
- Current deployment model is problematic and tightly coupled to legacy infrastructure
Goals
- replicate the existing production deployment environment as closely as practical within docker containers
- centos 6
- Apache 2.2
- PHP 5.3
- Python 2.7
- mod_wsgi 4.4
- starting from that base environment, develop and test environments that move away from legacy technologies
- alter apache config to replace mod_wsgi with proxy to gunicorn
- default to python instead of php
- eliminate php
- move all apache redirects to python
- replace apache with nginx
- replace mysql with postgresql
- add varnish cache
- full testing environment
- python unit tests
- javascript unit tests
- javascript behavior tests
  - casperjs
  - phantomjs headless webkit
  - slimerjs driving containerized firefox
- mcom-tests
  - python requests
  - python driving selenium full browser tests via
  - containerized firefox
  - saucelabs
- apache config testing
- Deployable to [[http://deis.io/][Deis]]
Non-goals
- replicating the current "Chief" deployment system within Docker containers
- production deployment of containerized apache
Quickstart
install docker
git clone https://github.com/jgmize/mozorg-env mozorg
- TODO: move this to mozilla github org
cd mozorg
add mozilla.local to /etc/hosts
- if using boot2docker, use the ip address from the command "boot2docker ip"
- on Linux, add it as an alias to 127.0.0.1
Bedrock development
- bin/start-bedrock-mysql
- wait for the "Ctrl+c to exit" line
- visit http://mozilla.local:8000 in your browser
Apache/PHP development
- bin/start-apache-mysql
- wait for the "Ctrl+c to exit" line
- edit apache/mozilla.com/includes/config.inc.php
- $config['server_name'] = 'mozilla.local';
- visit http://mozilla.local in your browser
Stop services
- bin/compose stop
- "Ctrl+c" does not currently stop services, only detaches the log output in the terminal
Access a bash shell inside a running container (or start container if necessary)
- bin/dsh bedrock
- bin/dsh apache
- bin/dsh mysql
Build and run a new image (only needed when adding new requirements)
- bin/start-apache-mysql -b
- bin/start-bedrock-mysql -b
Delete old files from boot2docker volumes
- bin/b2d-rsync --delete bedrock
- bin/b2d-rsync --delete apache/mozilla.com
- b2d-rsync script does not delete by default for performance reasons
Troubleshooting ** Hijack is incompatible with use of CloseNotifier
Try running the command again, it usually works the second time.
This is a known issue with docker-compose 1.2.0, see https://github.com/docker/docker/issues/12845 for more details.
This was fixed in https://github.com/docker/compose/pull/1374 but as of this writing docker-compose 1.3.0 has not been released
You can try installing the latest RC, or 1.3 may have been released since this document was updated: https://github.com/docker/compose/releases ** "Permission denied" OSerror for a css file whe running in apache and DEBUG=True
DEBUG should be set to False automatically for the apache container, but can be overridden
bedrock/settings/local.py is no longer needed, but supported for backwards compatibility
DEBUG will only work for the bedrock container ** Command prompt does not appear when running "bin/dsh bedrock" (or any other service)
hit enter again and the command prompt should appear
TODO: link to github issue ** PHP redirects to empty hostname
edit apache/mozilla.com/includes/config.inc.php
- $config['server_name'] = 'mozilla.local';
Continuous Integration and Deployment
[ ] Pull Requests
- [ ] Check whitelist
- [ ] no-op if not on whitelist (not even github comment)
- [ ] static analysis
- [ ] flake8
- [ ] jslint
- [ ] build docker image(s) if necessary
- [ ] base image
- [ ] dev image
- [ ] check for Dockerfile differences
  - [ ] exit 0 if no diff from prev build
- [ ] tag image with git commit
- [ ] push to docker hub
- [ ] peep install requirements
- [ ] docker commit and push to docker hub if needed
  - this will deploy new requirements as a layer on top of previous base image
  - maximizes cache utilization
  - this does not handle removing old requirements
  - that will happen on periodic rebuild of base image
- [ ] backend unit tests
- [ ] frontend tests
[ ] merge to master on bedrock
- [ ] tag base image(s) if necessary and push to docker hub
- [ ] deis dev deploys
- [ ] apache mod_wsgi + PHP + redirects
  - until all PHP and apache redirects are gone
- [ ] bedrock uWSGI
- [ ] mcom-tests against apache dev deployment
- [ ] mcom-tests against bedrock-only deploy can be manually triggered
- expected to fail until redirects ported from apache
- useful to verify during porting process
[ ] l10n merge to master or trunk
- [ ] deploy l10n updates to all environments
- additional layer in docker images on top of currently deployed images
[ ] product details update
- [ ] deploy product details updates to all environments
- additional layer in docker images on top of currently deployed images
- [ ] investigate moving product details to DB

jgmize / mozorg-env

readme