Clean up manage.py

[hou098]

See https://github.com/BioplatformsAustralia/bpaotu/blob/1.34.0/bpaotu/manage.py Is the in_production case actually used? For starters it has production_user = "apache" which I don't think is really the case. Also, ccg_django_utils.conf.setup_prod_env() needs prodsettings.py which doesn't seem to exist anywhere. See https://github.com/muccg/ccg-django-utils/blob/master/ccg_django_utils/conf.py#L43 Also, https://github.com/BioplatformsAustralia/bpaotu/blob/1.34.0/bpaotu/bpaotu/settings.py has plenty of env.get() calls which look like they're there to cater for runtime overrides in production.

[mtearle]

Hi @hou098

Here's what is in our puppet config that sets up the environment for the bpaotu container (with secrets redacted)

class bpaotu::prod { $otuversion = '1.34.0' $append_env = [ 'CSRF_TRUSTED_ORIGINS=".ccgapps.com.au"', 'CACHE="redis://redis:6379/1"', 'CELERY_BROKER_URL="redis://redis:6379/2"', 'REDIS_HOST=redis', 'REDIS_DB=2', 'DBNAME=prod_bpaotu', 'DBUSER=prod_bpaotu', 'DBPASS=REDACTED', 'DBSERVER=production.REDACTED.ap-southeast-2.rds.amazonaws.com', 'ALLOWED_HOSTS="data.bioplatforms.com"', 'CKAN_API_KEY="REDACTED-REDACTED-REDACTED-REDACTED-REDACTED"', 'BPAOTU_AUTH_SECRET_KEY=REDACTED', 'CKAN_AUTH_INTEGRATION=1', 'CSRF_COOKIE_DOMAIN=.data.bioplatforms.com', 'GALAXY_BASE_URL=https://usegalaxy.org.au', 'GALAXY_ADMIN_USER_API_KEY=REDACTED', 'GALAXY_KRONA_WORKFLOW_ID=REDACTED', 'BLAST_RESULTS_URL=/bpa/otu/blast-results/', 'AWS_ACCESS_KEY_FOR_ANYMAIL_SES="REDACTED"', 'AWS_SECRET_KEY_FOR_ANYMAIL_SES="REDACTED/REDACTED"', 'AWS_REGION_FOR_ANYMAIL_SES="us-east-1"', 'DEBUG=0', 'PRODUCTION=1']

dockersite { 'bpaotu': vhost => 'staging.ccgapps.com.au', localhost_port => '8101', secret_key => 'REDACTED', docker_image => 'bioplatformsaustralia/bpaotu', docker_links => ['redis:redis'], image_tag => $otuversion, location => 'bpaotu', append_env => $append_env, }

dockersite { 'bpaotuceleryworker': volume_name => 'bpaotu', docker_image => 'bioplatformsaustralia/bpaotu', localhost_port => '8102', docker_links => ['redis:redis'], image_tag => $otuversion, command => 'celery_worker', location => 'bpaotu', append_env => $append_env + ['DJANGO_SETTINGS_MODULE=bpaotu.settings', 'KEY_PREFIX=ccgbpaotu'], } }

[mtearle]

Looking inside the running containers on prod and staging (for both bpaotu and bpaotuceleryworker), no bpaotu.settings to be found

[mtearle]

These are the running processes "command lines" inside the bpaotu container on prod

bioplatforms@bpaotu:~$ for X in /proc/*/cmdline; do echo $X; cat $X; echo; done /proc/1/cmdline uwsgi--die-on-term--ini/app/uwsgi/docker.ini /proc/113/cmdline bash /proc/42/cmdline uwsgi--die-on-term--ini/app/uwsgi/docker.ini /proc/43/cmdline bpaotu-http-http-9000 /proc/44/cmdline bpaotu-socket-socket-9100 /proc/45/cmdline bpaotu-http-http-9000 /proc/46/cmdline bpaotu-http-http-9000 /proc/47/cmdline bpaotu-http-http-9000 /proc/48/cmdline bpaotu-http-http-9000 /proc/49/cmdline bpaotu-http-http-9000 /proc/50/cmdline bpaotu-socket-socket-9100 /proc/51/cmdline bpaotu-socket-socket-9100 /proc/52/cmdline bpaotu-socket-socket-9100 /proc/53/cmdline bpaotu-socket-socket-9100 /proc/self/cmdline cat/proc/self/cmdline /proc/thread-self/cmdline cat/proc/thread-self/cmdline

So just uwsgi related

[mtearle]

Perhaps the first stage of clean up here would be to add some logging on both execution paths of the if (true/false) before removing the code as appropriate

[hou098]

Thanks for that @mtearle . There's already some code there to log the production/development cases https://github.com/BioplatformsAustralia/bpaotu/blob/1.34.0/bpaotu/manage.py#L26

        print("{0} Django manage in production".format(PROG))

Anything like

bpaotu Django manage in production

in the runserver container logs?

[mtearle]

ubuntu@bpa-prod:~$ docker logs -tf bpaotu 2>&1 | grep -i django 2022-05-24T23:54:01.737950593Z DBDRIVER=django.db.backends.postgresql_psycopg2 2022-05-24T23:54:01.737978307Z DJANGO_FIXTURES=none 2022-05-24T23:54:01.737982184Z DJANGO_SETTINGS_MODULE=bpaotu.settings 2022-05-24T23:54:01.754155184Z + django-admin.py collectstatic --noinput --settings=bpaotu.settings 2022-05-24T23:54:02.927057059Z + django-admin.py migrate --noinput --settings=bpaotu.settings 2022-05-24T23:54:05.579661024Z + django-admin.py warmcache --settings=bpaotu.settings 2022-05-25T01:30:11.923334891Z + django-admin.py check --deploy --settings=bpaotu.settings

ubuntu@bpa-prod:~$ docker logs -tf bpaotuceleryworker 2>&1 | grep -i django 2022-05-24T23:54:00.629099656Z DBDRIVER=django.db.backends.postgresql_psycopg2 2022-05-24T23:54:00.629122678Z DJANGO_FIXTURES=none 2022-05-24T23:54:00.629125260Z DJANGO_SETTINGS_MODULE=bpaotu.settings

Nothing of note in the registry.log file for either of them

[hou098]

Ah yes, now that I think about it, manage.py isn't invoked during production webserver operation. I think it uses django.wsgi See https://github.com/BioplatformsAustralia/bpaotu/blob/1.34.0/uwsgi/vassals/socket-9100.ini#L17

I think I might file this issue under "ain't broke don't fix it" for now.