Base API functionality

[kiniadit]

Docker

• switched from env.sh to .env to handle environment variables. Possibly more scalable
• created two separate Dockerfiles - DOCKERFILE.db and DOCKERFILE.api for postgres and api containers repsectively

Django

• Models for crash, partic and vhcl (tables from the ODOT Postgres db) now working
• Corresponding API endpoints now accessible via GET requests
• pagination support added

API ROOT is accessible from - http://:8000/api/ crash tables are accessible from - http://:8000/api/crashes/ partic tables are accessible from - http://:8000/api/participants/ vhcl tables are accessible from - http://:8000/api/vehicles/

[znmeb]

Test results:

1. I had to "export" the environment variables in the compose file: POSTGRES_PASSWORD, PROJECT_NAME and DEBUG. I didn't know what to use for DEBUG so I used "false".
2. I had to define a file bin/.env. I put all the variables I knew about.
3. I had to copy odot_crash_data.sql into bin/.

Then I did a docker-compose build and a docker-compose up -d. The database container comes up but the API container crashed:

 $ docker logs transportationsystembackend_api_1 
Traceback (most recent call last):
  File "./manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 62, in execute
    super().execute(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 71, in handle
    if not settings.DEBUG and not settings.ALLOWED_HOSTS:
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 43, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 125, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

So I defined SECRET_KEY both as an export and in bin/.env. It crashed the same way, so I looked at the code and discovered I need to set DJANGO_SECRET_KEY. So I did that and now the API container stays up.

I can browse to it now but it is resetting. It's having trouble connecting to the database. Your connection string needs to connect to host "db" as user "postgres" on port 5432 with the password $POSTGRES_PASSWORD. It looks like it's trying to connect to a database inside the Django API container and there isn't one running there.

docker logs -f transportationsystembackend_api_1 
Performing system checks...

System check identified no issues (0 silenced).
Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x7fe0f3d35950>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 168, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/utils/autoreload.py", line 225, in wrapper
    fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 124, in inner_run
    self.check_migrations()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 427, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 49, in __init__
    self.build_graph()
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 206, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 61, in applied_migrations
    if self.has_table():
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 44, in has_table
    return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 255, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 232, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 168, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

It looks like the only thing that needs to be fixed is the connection string to the database container. Everything else we can get in the documentation cleanup.

[BrianHGrant]

DEBUG connects to the "DEBUG_MODE" in the settings.py ATM, which should turn the dev server on or off. My intent in including was that it might be able to be wired up to other areas will you might want such a switch, one variable to rule them all.

I'm fairly sure the host name for the dstabase in settings.py should be the name of the database container in the docker-compose, which was 'db', in the version I wrote.

In this case it looks like it is being set to a OS variable name 'db', which you may or may not have set to the container name. If not set it would default to localhost, which would be inside the container.

I have not tried this as of yet, but think may be the issue.

On Feb 5, 2018 10:39 PM, "M. Edward (Ed) Borasky" notifications@github.com wrote:

Test results:

1. I had to "export" the environment variables in the compose file: POSTGRES_PASSWORD, PROJECT_NAME and DEBUG. I didn't know what to use for DEBUG so I used "false".
2. I had to define a file bin/.env. I put all the variables I knew about.
3. I had to copy odot_crash_data.sql into bin/.

Then I did a docker-compose build and a docker-compose up -d. The database container comes up but the API container crashed:

$ docker logs transportationsystembackend_api_1 Traceback (most recent call last): File "./manage.py", line 15, in execute_from_command_line(sys.argv) File "/usr/local/lib/python3.6/site-packages/django/core/management/init.py", line 371, in execute_from_command_line utility.execute() File "/usr/local/lib/python3.6/site-packages/django/core/management/init.py", line 365, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv self.execute(*args, cmd_options) File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 62, in execute super().execute(*args, *options) File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute output = self.handle(args, options) File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 71, in handle if not settings.DEBUG and not settings.ALLOWED_HOSTS: File "/usr/local/lib/python3.6/site-packages/django/conf/init.py", line 56, in getattr self._setup(name) File "/usr/local/lib/python3.6/site-packages/django/conf/init.py", line 43, in _setup self._wrapped = Settings(settings_module) File "/usr/local/lib/python3.6/site-packages/django/conf/init.py", line 125, in init raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.") django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

So I defined SECRET_KEY both as an export and in bin/.env. It crashed the same way, so I looked at the code and discovered I need to set DJANGO_SECRET_KEY. So I did that and now the API container stays up.

I can browse to it now but it is resetting. It's having trouble connecting to the database. Your connection string needs to connect to host "db" as user "postgres" on port 5432 with the password $POSTGRES_PASSWORD. It looks like it's trying to connect to a database inside the Django API container and there isn't one running there.

docker logs -f transportationsystembackend_api_1 Performing system checks...

System check identified no issues (0 silenced). Unhandled exception in thread started by <function check_errors..wrapper at 0x7fe0f3d35950> Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection self.connect() File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect self.connection = self.get_new_connection(conn_params) File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 168, in get_new_connection connection = Database.connect(conn_params) File "/usr/local/lib/python3.6/site-packages/psycopg2/init.py", line 130, in connect conn = _connect(dsn, connection_factory=connection_factory, kwasync) psycopg2.OperationalError: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/django/utils/autoreload.py", line 225, in wrapper fn(*args, kwargs) File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 124, in inner_run self.check_migrations() File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 427, in check_migrations executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS]) File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 18, in init self.loader = MigrationLoader(self.connection) File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 49, in init self.build_graph() File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 206, in build_graph self.applied_migrations = recorder.applied_migrations() File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 61, in applied_migrations if self.has_table(): File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 44, in has_table return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()) File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 255, in cursor return self._cursor() File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 232, in _cursor self.ensure_connection() File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection self.connect() File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in exit raise dj_exc_value.with_traceback(traceback) from exc_value File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection self.connect() File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect self.connection = self.get_new_connection(conn_params) File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 168, in get_new_connection connection = Database.connect(conn_params) File "/usr/local/lib/python3.6/site-packages/psycopg2/init.py", line 130, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) django.db.utils.OperationalError: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hackoregon/transportation-system-backend/pull/14#issuecomment-363325906, or mute the thread https://github.com/notifications/unsubscribe-auth/ARHAUUk1lPunFVfmeE6H3rbRPSMXDEb8ks5tR_OngaJpZM4R6M5u .

[znmeb]

@BrianHGrant What about the merge conflicts GitHub flagged? I don't know what to do about those. But I'll go ahead and try the settings.py change on my machine to see what happens.

It turns out it's another environment variable - POSTGRES_NAME. So it looks like it's just documentation cleanup now.

It's still not connecting - I guess we'll have to wait till tomorrow.

[znmeb]

Another documentation issue - as the code stands now the tables are restored to the postgres database. That will work for the current setup as long as the Django client code knows to look for them there. If we have a single container serving multiple databases we'll want a database for each dataset though.

[BrianHGrant]

We'll take some time at meetin to resolve the merge conflicts.

Our basic practice for setting up the compose, variables, and database should roughly follow the steps here: https://docs.docker.com/compose/django/

We should try to uncomplicate the config and use of variables if it is becoming too much to follow, @kiniadit if you could include a sample '.env' with redacted with info in repo it would be useful, something like .env.sample

[znmeb]

@BrianHGrant Ah - that explains why the Django container is on "jessie" and not "stretch" - the default Python 3 is still on "jessie" (https://store.docker.com/images/python). We can probably live with mixed Debian releases but we'll definitely want to keep the PostgreSQL client and server libraries at the same release levels. I have a way to do that but it'll take me a few hours to make sure it's working. So it will have to wait till this pull request is merged and we have some connection test code.

[BrianHGrant]

closing this PR do to being on master. please reopen to staging.