Clojure failing on db-url assertion using docker-compose

[rohanrichards]

I am getting the following error running docker-compose up with no other configuration.

polis-math  | Caused by: java.lang.AssertionError: Assert failed: Missing database url. Make sure to set env variables.
polis-math  | database-url

Here is the relevant code from polisMath/src/posmath/components/postgres.clj

(defrecord Postgres [config db-spec]
  component/Lifecycle
  (start [component]
    (log/info ">> Starting Postgres component")
    (let [database-url (-> config :database :url)]
      (assert database-url "Missing database url. Make sure to set env variables.")
      (assoc component :db-spec (heroku-db-spec database-url))))
  (stop [component]
    (log/info "<< Stopping Postgres    component")
    (assoc component :db-spec nil)))

As far as I can tell the environment variables are being set, yet I don't know Clojure at all so am struggling to decipher the above, and whether its relevant that the variable set by docker is called DATABASE_URL and not database-url

I will keep digging into polisMath and do a bit of reading about Clojure to see if I can shed any light on this.

[rohanrichards]

I noticed there was a docker-dev.env file that wasn't in use so I modified some docker-compose.yml containers to make use of it:

  math:
    container_name: polis-math
    env_file: docker-dev.env
    depends_on:
      - 'postgres'
    build:
      context: https://github.com/pol-is/polisMath.git

This has allowed me to progress beyond the above assertion error, but it looks like polis-math is still having trouble connecting to postgres:

polis-math  | 19-01-10 01:18:36 9b5389ca83b0 INFO [polismath.poller:23] - Polling :moderation > 1546219113413
polis-math  | 19-01-10 01:18:36 9b5389ca83b0 INFO [polismath.components.postgres:116] - modpoll 1546219113413
polis-math  | 19-01-10 01:18:36 9b5389ca83b0 ERROR [polismath.components.postgres:124] - moderation polling failed  The connection attempt failed.
polis-math  | 19-01-10 01:18:36 9b5389ca83b0 INFO [polismath.poller:23] - Polling :votes > 1546219113426
polis-math  | 19-01-10 01:18:36 9b5389ca83b0 INFO [polismath.components.postgres:100] - poll 1546219113426
polis-math  | 19-01-10 01:18:36 9b5389ca83b0 ERROR [polismath.components.postgres:107] - polling failed  The connection attempt failed.
polis-math  | org.postgresql.util.PSQLException: The connection attempt failed.

[rohanrichards]

Adding

    networks:
      - 'polis-dev'

to the math container now gives org.postgresql.util.PSQLException: The server does not support SSL. I will continue digging!

[rohanrichards]

I figure its safe to disable SSL on the connections internal to docker, so I've changed the DATABASE_URL environment variable inside of docker-dev.env (which I now hand over to all the containers) to this DATABASE_URL=postgres://postgres:oiPorg3Nrz0yqDLE@postgres:5432/polis-dev?sslmode=disable The last error is gone, and polis-math now logs this:

polis-math  | 19-01-10 01:49:24 b51740ee4702 DEBUG [polismath.tasks:69] - polling tasks from 1546220940746
polis-math  | 19-01-10 01:49:24 b51740ee4702 DEBUG [polismath.tasks:73] - new last-timestamp 1546220940746
polis-math  | 19-01-10 01:49:24 b51740ee4702 INFO [polismath.poller:23] - Polling :votes > 1546220940753
polis-math  | 19-01-10 01:49:24 b51740ee4702 INFO [polismath.components.postgres:100] - poll 1546220940753
polis-math  | 19-01-10 01:49:25 b51740ee4702 INFO [polismath.poller:23] - Polling :moderation > 1546220940741
polis-math  | 19-01-10 01:49:25 b51740ee4702 INFO [polismath.components.postgres:116] - modpoll 1546220940741
polis-math  | 19-01-10 01:49:25 b51740ee4702 DEBUG [polismath.tasks:80] - post timeout

As I've never gotten pol-is running before, I dont know if this is expected behavior...

Assuming the above is indeed healthy, all that's left is actually loading something up at localhost:5000 which fails due to https://github.com/pol-is/polis-deploy/issues/4 so this can probably be closed.

[ballPointPenguin]

Good stuff. I took a stab at rethinking how we use Docker with PolisMath, but had not finished that work. I will revisit: https://github.com/ballPointPenguin/polisMath/commit/02bcf5c7df3b036977b793a8467a72e4c07ccd1d

[metasoarous]

@rohanrichards Thanks for sorting through the SSL issue; looks like simply adding sslmode=disable solves the problem? IIRC this was actually where we got stopped the last time we were working on this, so nice to see a resolution there. I'm thinking we add a config variable which determines whether the postgresql db should be connected to with SSL or not.

Looking at the logs, my guess is that the database still isn't connected or set up properly, or the polling would repeat at a regular interval, and the post timeout message adds to my suspicion.

[rohanrichards]

@metasoarous yes that looks like the fix for the SSL error, I think you'd need a different docker base image (postgres-ssl) to get SSL actually working though. However I feel because all communication is internal to the docker instance it's not an issue though? I'm really not sure how to debug this further, but I'll tinker with @ballPointPenguin 's commit above and try to dig deeper into the postgres service to see if I can find what's going wrong.

[metasoarous]

My understanding is that for the simplest deployment, it's correct that you wouldn't really need SSL. I'm a little less sure about whether that's the case if you wanted to run the database on a separate machine. But this kind of more involved deployment is not the main concern, and I think sslmode=disable is fine for now as far as the docker setup is concerned. Still, I'd rather leave the math code itself a bit flexible as to how things are being deployed, and so an optional environment setting here seems appropriate and easy enough to accommodate.

I'm still not sure why you might be getting stuck where you are, but one thing worth trying would be to see if you can connect via the pg CLI to the database container, and maybe poke around to see if the schema/tables look like the setup in polisServer/postgres.

Thanks again!

[rohanrichards]

I got polisClientParticipation building and have created a PR. I will go back to digging around in the postgres tables as suggested, failing that I will work on the clientAdmin Dockerfile next.

[rohanrichards]

@metasoarous I connected to the polis-db container and poked about in postgres, the first thing I noticed is when I try to connect with psql I get a "user root does not exist" type error. After a bit of reading it looks like postgres prefers if the username and the db name match. This could be a hint? To get psql to work I have to add a --username flag; psql --username postgres is this flag being used in appropriate locations in polisMath? I can't answer that currently...I will keep digging. On a related note, once I connect to the db and list the tables, everything looks pretty normal and seems to match the schema in polisServer/postgres/db_setup_draft.sql

[metasoarous]

In polismath, it's parsing it out of the database url (in hereoku format). The psql command may not handle the url the same way.

[patcon]

@metasoarous seems that the ignore-ssl config flag was removed: dfe233e8

Is this ok to leave disabled, or should there be a way to enable for prod? If nothing needs change or discussion, then I guess we could close this issue, unless anyone objects

[patcon]

I think we're good to close this for now :) Can always revert if I'm mistaken!