Travis builds do not succeed

[MikeTheCanuck]

According to the [last Travis build](), the build is failing at the project_config import:

  File "/code/transDjango/transDjango/settings.py", line 14, in <module>
    from .settings_local import *
ImportError: No module named 'transDjango.settings_local'

At line 2016 of that build log we see why: ./bin/getconfig.sh: line 18: aws: command not found

Next step is to get the awscli to install correctly.

[MikeTheCanuck]

Progress:

• added the awscli install to Dockerfile and that got the command to run in build 16
• added a project_config.py to the hacko-transportation-config S3 bucket (there was none there previously) - NOTE: this one points to a database server that's shared with Budget, so we'll have to track this down and separate them someday soon
• in build 17 we now definitely have the project_config.py downloading successfully during the build

Strangely, the settings.py file contains this line: from .settings_local import *

I'm mystified how the app is working with an import like that - I can't find a .settings_local folder anywhere in the GitHub repo. I see the settings_local_example.py and it has all the settings that connect to the databases etc, but I don't have any good ideas at midnight how to adapt this to (a) work for a Travis build (where should Travis get a copy of this file?) and (b) can postgresql substitute for postgis?

Either way, once we have these worked out, the new settings file will have to be configured to pull in the project_config.py secrets.

[MikeTheCanuck]

I muddled around for a while, trying to reconstruct settings.py, urls.py and disabling all the 'db'-related dependencies in the docker-compose files, so that we could (a) migrate and populate the DB to the EC2-hosted PostgreSQL, and (b) get a buildable container that follows the general pattern the rest of the Django projects do.

However, there were a lot of build failures during container startup, and eventually I gave up trying to figure out what's going on in the Django app. Once there's a working app that can configure and use a remote DB in EC2, getting a deployable container seems like a great idea; until then, I'll keep running into problems such as this:

$ bin/start-proj.sh -l
Starting transportationbackend_transport-service_1
Attaching to transportationbackend_transport-service_1
transport-service_1  | Traceback (most recent call last):
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
transport-service_1  |     return self.cursor.execute(sql)
transport-service_1  | psycopg2.ProgrammingError: permission denied to create extension "postgis"
transport-service_1  | HINT:  Must be superuser to create this extension.
transport-service_1  | 
transport-service_1  | 
transport-service_1  | The above exception was the direct cause of the following exception:
transport-service_1  | 
transport-service_1  | Traceback (most recent call last):
transport-service_1  |   File "/code/transDjango/manage.py", line 22, in <module>
transport-service_1  |     execute_from_command_line(sys.argv)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
transport-service_1  |     utility.execute()
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
transport-service_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
transport-service_1  |     self.execute(*args, **cmd_options)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
transport-service_1  |     output = self.handle(*args, **options)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 81, in handle
transport-service_1  |     connection.prepare_database()
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 25, in prepare_database
transport-service_1  |     cursor.execute("CREATE EXTENSION IF NOT EXISTS postgis")
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 79, in execute
transport-service_1  |     return super(CursorDebugWrapper, self).execute(sql, params)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
transport-service_1  |     return self.cursor.execute(sql, params)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
transport-service_1  |     six.reraise(dj_exc_type, dj_exc_value, traceback)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
transport-service_1  |     raise value.with_traceback(tb)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
transport-service_1  |     return self.cursor.execute(sql)
transport-service_1  | django.db.utils.ProgrammingError: permission denied to create extension "postgis"
transport-service_1  | HINT:  Must be superuser to create this extension.
transport-service_1  | 
transport-service_1  | Traceback (most recent call last):
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
transport-service_1  |     return self.cursor.execute(sql, params)
transport-service_1  | psycopg2.ProgrammingError: relation "APIimports_apielement" does not exist
transport-service_1  | LINE 1: SELECT "APIimports_apielement"."apiName" FROM "APIimports_ap...
transport-service_1  |                                                       ^
transport-service_1  | 
transport-service_1  | 
transport-service_1  | The above exception was the direct cause of the following exception:
transport-service_1  | 
transport-service_1  | Traceback (most recent call last):
transport-service_1  |   File "/code/transDjango/manage.py", line 22, in <module>
transport-service_1  |     execute_from_command_line(sys.argv)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
transport-service_1  |     utility.execute()
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
transport-service_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
transport-service_1  |     self.execute(*args, **cmd_options)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
transport-service_1  |     output = self.handle(*args, **options)
transport-service_1  |   File "/code/transDjango/APIimports/management/commands/import_jsons.py", line 14, in handle
transport-service_1  |     oneRingToBindThem()
transport-service_1  |   File "/code/transDjango/APIimports/importers/get_data.py", line 18, in oneRingToBindThem
transport-service_1  |     if name in list(ApiElement.objects.values_list('apiName', flat=True)):
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 256, in __iter__
transport-service_1  |     self._fetch_all()
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 1087, in _fetch_all
transport-service_1  |     self._result_cache = list(self.iterator())
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 155, in __iter__
transport-service_1  |     for row in compiler.results_iter():
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 789, in results_iter
transport-service_1  |     results = self.execute_sql(MULTI)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
transport-service_1  |     cursor.execute(sql, params)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 79, in execute
transport-service_1  |     return super(CursorDebugWrapper, self).execute(sql, params)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
transport-service_1  |     return self.cursor.execute(sql, params)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
transport-service_1  |     six.reraise(dj_exc_type, dj_exc_value, traceback)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
transport-service_1  |     raise value.with_traceback(tb)
transport-service_1  |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
transport-service_1  |     return self.cursor.execute(sql, params)
transport-service_1  | django.db.utils.ProgrammingError: relation "APIimports_apielement" does not exist
transport-service_1  | LINE 1: SELECT "APIimports_apielement"."apiName" FROM "APIimports_ap...
transport-service_1  |                                                       ^
transport-service_1  | 
transport-service_1  | Unknown command: 'jsonToCIPPoints'
transport-service_1  | Type 'manage.py help' for usage.
transport-service_1  | Unknown command: 'jsonToCIPLines'
transport-service_1  | Type 'manage.py help' for usage.
transport-service_1  | Unknown command: 'jsonToCIPPolygons'
transport-service_1  | Type 'manage.py help' for usage.
transport-service_1  | Performing system checks...
transport-service_1  | 
transport-service_1  | System check identified no issues (0 silenced).
transport-service_1  | 
transport-service_1  | You have 31 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): APIimports, admin, auth, contenttypes, sessions.
transport-service_1  | Run 'python manage.py migrate' to apply them.

Maybe I need to start over, but this is both deflating and frustrating.

[znmeb]

I don't know if Transportation can use a remote PostgreSQL database - we have our own geocoder https://github.com/hackoregon/postgis-geocoder-test. I've heard that @BrianHGrant has all this magic working for Emergency response, including the geocoder. Anything I can do to help? I want to get this working ASAP - we're falling behind the other teams and I've got loads of free time.

[MikeTheCanuck]

Hi @znmeb - I don't know enough about the geocoder dependency, but I'm assuming that the Transportation APIs also have some basic PostgreSQL requirement as well - and in one of the code branches I see this command in a Dockerfile for the "db" layer: RUN apt-get -y install postgresql-9.5-postgis-2.2.

Is that an accurate representation, or is the entirety of Transportation's backend data stored in a non-Postgres enivronment?

[znmeb]

All of the data's in PostgreSQL but the geocoder needs PostgreSQL 9.6 / PostGIS 2.3 - see repo https://github.com/hackoregon/postgis-geocoder-test.

Note that I'm working on a build that includes virtualenvwrappper and thus should run any Django project against the same database. I'm doing that mostly to simplify my own local testing but if it's useful to other projects they can use it. I can put Travis on it as soon as I learn how. ;-)

[MikeTheCanuck]

As of Travis build 39 the deploy pipeline is successfully delivering the Django app in containerized form to AWS ECS.