Migrations don't work on Heroku

[duncan-bayne]

I've successfully deployed a Luminus 3.25 application to Heroku.

If I migrate the database locally, the migration succeeds:

2019-04-07 17:58:50,383 [main] DEBUG org.jboss.logging - Logging Provider: org.jboss.logging.Slf4jLoggerProvider
2019-04-07 17:58:52,256 [main] INFO  migratus.core - Starting migrations
2019-04-07 17:58:52,530 [main] INFO  migratus.database - creating migration table 'schema_migrations'
2019-04-07 17:58:52,574 [main] DEBUG migratus.migrations - Looking for migrations in #object[java.io.File 0xf24a38a /usr/home/duncan/code/todo-list/resources/migrations]
2019-04-07 17:58:52,598 [main] INFO  migratus.core - Running up for [20190407152249]
2019-04-07 17:58:52,600 [main] INFO  migratus.core - Up 20190407152249-add-users-table
2019-04-07 17:58:52,635 [main] DEBUG migratus.migration.sql - found 1 up migrations
2019-04-07 17:58:52,642 [main] DEBUG migratus.database - marking 20190407152249 complete
2019-04-07 17:58:52,656 [main] INFO  migratus.core - Ending migrations

But if I migrate the Heroku database as per the documentation, it fails:

Running lein run migrate on heroku-app-name... starting, run.8339 (Free)
Running lein run migrate on heroku-app-name... connecting, run.8339 (Free)
Running lein run migrate on heroku-app-name... up, run.8339 (Free)
... snip ...
Retrieving ring/ring-mock/0.3.2/ring-mock-0.3.2.jar from clojars
2019-04-07 07:50:44,110 [main] DEBUG org.jboss.logging - Logging Provider: org.jboss.logging.Slf4jLoggerProvider
Exception in thread "main" Syntax error compiling at (/tmp/form-init7943197157403759538.clj:1:73).
  at clojure.lang.Compiler.load(Compiler.java:7647)
  ... snip ...
  at clojure.main.main(main.java:37)
Caused by: java.lang.RuntimeException: could not start [#'todo-list.config/env] due to
  at mount.core$up$fn__285.invoke(core.cljc:80)
  ... snip ...
  Caused by: java.lang.RuntimeException: could not find a non empty configuration file to load. looked in the classpath (as a "resource") and on a file system via "conf" system property
  ... snip ...

[yogthos]

It looks like there is some missing configuration according to the exception. I haven't used Heroku in a while myself, so I'm not sure what the cause is specifically. Have you tried the approach of running the migrations directly from the app on start up?

[duncan-bayne]

@yogthos I haven't - I'm actually fairly new to Clojure; I've done more Common Lisp (and in fact have worked on a Common Lisp buildpack for Heroku). I'll continue experimenting and see what I can figure out. Will post updates here.

[yogthos]

sounds good 👍

[duncan-bayne]

Things I've tried so far:

• Adding a no-op production-config.edn file.
• Adding a no-op prod-config.edn file.
• Adding a Heroku environment variable CONF, pointing to env/prod/resources/config.edn.
• Changing CONF to point to an absolute pathname /env/prod/resources/config.edn.

None of the above work. It's not clear to me where cprop is looking for configuration, or how to debug it. :thinking: I think I need a proper debugging approach, not spray & pray / guesswork.

If I were in CL + EC2 land, I'd a) set up an SSH tunnel into the box, b) fire up SLIME, c) load the app and wait to drop into the debugger. I might try a similar approach w/ Clojure, assuming I can get Cider connected to the Heroku dyno.

[duncan-bayne]

Further digging:

1. The uberjar prepared during Heroku deployment contains config.edn, which is the production configuration.
2. Logging in to a Heroku dyno and running lein run migrate manually produces the same failure.

[duncan-bayne]

lein repl on a Heroku dyno times out, too. So not much help there.

[duncan-bayne]

A run w/ DEBUG=y produces the following:

Leiningen's classpath: /app/.lein/self-installs/leiningen-2.8.3-standalone.jar                                                                                            Applying task run to [migrate]
Applying task javac to nil                                  
Running javac with [@/tmp/.leiningen-cmdline4251380810423337457.tmp]
Applying task compile to nil                          
All namespaces already AOT compiled.                            
2019-04-08 05:17:41,392 [main] DEBUG org.jboss.logging - Logging Provider: org.jboss.logging.Slf4jLoggerProvider
(!) read config from resource: "config.edn", but it is empty

Next step (heh) is to figure out where in my project there is an empty EDN.

[duncan-bayne]

\o/ Okay, that worked:

2019-04-08 05:33:46,538 [main] INFO  migratus.core - Starting migrations 
2019-04-08 05:33:46,957 [main] INFO  migratus.database - creating migration table 'schema_migrations' 
2019-04-08 05:33:47,088 [main] DEBUG migratus.migrations - Looking for migrations in #object[java.io.File 0x19778839 /app/resources/migrations] 
2019-04-08 05:33:47,105 [main] INFO  migratus.core - Running up for [20190407152249] 
2019-04-08 05:33:47,106 [main] INFO  migratus.core - Up 20190407152249-add-users-table 
2019-04-08 05:33:47,137 [main] DEBUG migratus.migration.sql - found 1 up migrations 
2019-04-08 05:33:47,149 [main] DEBUG migratus.database - marking 20190407152249 complete 
2019-04-08 05:33:47,167 [main] INFO  migratus.core - Ending migrations 

... but I'm mildly concerned, because this suggests that either the dev or test EDN is making its way into the Uberjar. Will investigate which is.

[duncan-bayne]

Okay! No, it was the production EDN, which is nice. In the end, the fix required to make migrations work on Heroku is to make env/prod/resources/config.edn contain:

;; prod/config.edn
{:prod true
 :port 3000}

@yogthos Should this be changed in the default Luminus template? Either that, or we should make a note in the documentation.

[yogthos]

Glad to see you got things working. I think it's probably better to add a note regarding the config in the deployment section since it's Heroku specific.

As a note, project.clj profiles are set up to use env/dev in dev, and env/prod for uberjar preventing dev/test environment from leaking into prod by design. And the config gets resolved by looking at a packaged one first, then merging an external config file when conf environment variable is specified, then merging any variables from the environment on top of that. I typically use Docker for deployments, so I just package a config.edn in the container.

[yogthos]

Could you try omitting :port in your config, that should come from the environment. For running migrations it shouldn't be needed at all since the HTTP server doesn't need to be started.

[duncan-bayne]

@yogthos Thanks, I'll remove :port. Shall I raise a PR on the Luminus docs for Heroku deployment?

[yogthos]

I added this bit earlier, but feel free to expand on it.

[duncan-bayne]

No that's great @yogthos, thanks :)

[D00mch]

I just created a project with luminus, and in the project.clj file there is a form,

  {:uberjar {:omit-source true
             :aot :all
             :uberjar-name "imagelang.jar"
             :source-paths ["env/prod/clj" ]
             :resource-paths ["env/prod/resources"]}

meaning that generated jar file will check the config.edn in the env/prod/resources (the only place for env/prod/resources), and it seems that

heroku run lein run migrate

from the documentation is not going to work.

Migration could be done running the generated jar file:

heroku run java -cp target/uberjar/<app name>.jar clojure.main -m imagelang.core migrate