Migratus doesn't see any migrations with Java 9

[pasih]

Hey,

I'm struggling to get Migratus to work. Now, I should mention that I'm pretty new to Clojure and this is more likely me failing at this than an actual bug.

The problem is that migratus pending and migratus migrate don't see any migrations.

I have [migratus "1.0.1"] in my project.clj dependencies and a db configured:

:migratus {:store :database
             :migration-dir "migrations/"
             :db {:subprotocol "mysql"
                  :subname "//127.0.0.1:3310/dbname"
                  :user "dbuser"
                  :password "dbpass"}})

The database connection details are correct (the port's non-standard but correct), I can connect to the db manually and, importantly, migratus is able to create the schema_migrations table so it seems to be able to connect too (the table is empty though).

I've got the lein plugin installed too: plugins [[migratus-lein "0.5.3"]].

lein migratus create foo creates migrations in the correct directory (resources/migrations) and my contents should be fine too (I've tried different contents):

  create table foo (id int not null auto_increment primary key);

Despite all this, lein migratus just gives me:

 migrating all outstanding migrations
17-12-14 07:17:11 UnknownHost INFO [migratus.core:288] - Starting migrations
Thu Dec 14 09:17:16 EET 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
17-12-14 07:17:16 UnknownHost INFO [migratus.core:288] - Ending migrations

lein migratus pending:

isting pending migrations
Thu Dec 14 09:17:37 EET 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
You have 0 pending migrations:

I've tried quite a few things but I just can't get Migratus to pick up my migrations.

Here's full list of my deps:

:dependencies [[org.clojure/clojure "1.8.0"]
                 [com.walmartlabs/lacinia "0.23.0"]
                 [com.walmartlabs/lacinia-pedestal "0.5.0"]
                 [clj-http "2.3.0"]
                 [org.clojure/data.json "0.2.6"]
                 [org.clojure/data.xml "0.0.8"]
                 [migratus "1.0.1"]
                 [mysql/mysql-connector-java "5.1.45"]
                 [com.fzakaria/slf4j-timbre "0.3.7"]
                 [io.pedestal/pedestal.service "0.5.2"]
                 [io.pedestal/pedestal.jetty "0.5.2"]]

[yogthos]

HI,

The steps look correct, and I was able to run migrations locally in a sample project against Postgres in a test project just now. You should be seeing something like the following:

lein migratus create foo
creating migration files for foo
nil

I end up with the two migration files created

ls resources/migrations/
20171214092826-foo.down.sql 20171214092826-foo.up.sql

I updated the up migration with create table foo (id serial primary key); and then ran the plugin where it created the schema_migrations table and applied the migration.

lein migratus
migrating all outstanding migrations
Dec 14, 2017 9:42:51 AM clojure.tools.logging$eval293$fn__297 invoke
INFO: Starting migrations
Dec 14, 2017 9:42:51 AM clojure.tools.logging$eval293$fn__297 invoke
INFO: creating migration table 'schema_migrations'
Dec 14, 2017 9:42:51 AM clojure.tools.logging$eval293$fn__297 invoke
INFO: Running up for [20171214092826]
Dec 14, 2017 9:42:51 AM clojure.tools.logging$eval293$fn__297 invoke
INFO: Up 20171214092826-foo
Dec 14, 2017 9:42:51 AM clojure.tools.logging$eval293$fn__297 invoke
INFO: Ending migrations

One thing to check would be if there's already an existingschema_migrations table in the db.

[pasih]

I'm using MySQL if that's relevant. Like I said, the weird thing is that it does create the schema_migrations table (i.e. it can access the database).

If I delete the table and re-run lein migratus, I get:

migrating all outstanding migrations
17-12-15 06:59:42 UnknownHost INFO [migratus.core:288] - Starting migrations
Fri Dec 15 08:59:48 EET 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
17-12-15 06:59:48 UnknownHost INFO [migratus.database:288] - creating migration table 'schema_migrations'
17-12-15 06:59:48 UnknownHost INFO [migratus.core:288] - Ending migrations

The schema_migrations table is still empty but no migrations are applied. I guess I'll have to do some debugging.

[yogthos]

Yeah that's definitely odd, and there shouldn't be anything MySQL specific about this. Could you try this test project to see if you have the same problem. It's setup for Postgres, but you can just switch the config for MySQL and it should work.

[pasih]

Exact same result. The table (schema_migrations, not 'foo') gets created with no content.

migrating all outstanding migrations
Dec 15, 2017 4:29:32 PM clojure.tools.logging$eval293$fn__297 invoke
INFO: Starting migrations
Fri Dec 15 16:29:33 EET 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Dec 15, 2017 4:29:33 PM clojure.tools.logging$eval293$fn__297 invoke
INFO: creating migration table 'schema_migrations'
Dec 15, 2017 4:29:33 PM clojure.tools.logging$eval293$fn__297 invoke
INFO: Ending migrations

(Only change I made was adding [mysql/mysql-connector-java "5.1.45"] and removing the PQL one and obviously adjusting the connection string - which I believe should be right because it manages to create schema_migrations. And the user definitely has insert rights)

[yogthos]

Perhaps this is something MySQL specific, I'll take a look at running the project with MySQL in a bit. I can't think of anything that should be database specific that would affect what migrations get run though.

[yogthos]

I got a chance to try this with MySQL 5.6.23 on another machine, and my migrations ran:

lein migratus
migrating all outstanding migrations
Dec 15, 2017 4:57:49 PM clojure.tools.logging$eval293$fn__297 invoke
INFO: Starting migrations
Dec 15, 2017 4:57:50 PM clojure.tools.logging$eval293$fn__297 invoke
INFO: creating migration table 'schema_migrations'
Dec 15, 2017 4:57:50 PM clojure.tools.logging$eval293$fn__297 invoke
INFO: Running up for [20171214092826]
Dec 15, 2017 4:57:50 PM clojure.tools.logging$eval293$fn__297 invoke
INFO: Up 20171214092826-foo
Dec 15, 2017 4:57:50 PM clojure.tools.logging$eval293$fn__297 invoke
INFO: Ending migrations

Here's the config I used:

:migratus
    {:store :database
     :migration-dir "migrations/"
     :db {:subprotocol "mysql"
          :subname "//127.0.0.1:3306/myapp_test"
          :user "root"
          :password "root"}}

[pasih]

I actually suspect it's not DB-related at all. I think it might simply not find the migration files (the directory should be correct though - after all they're created in the correct directory).

I'll try to debug this a bit when I have time. Maybe run Migratus' tests on this laptop.

(MacOS / HFS)

[yogthos]

Yeah I agree, it this doesn't sound like it's a db issue. I'm on MacOS/HFS as well, so not sure what's different. If you do find out definitely let me know.

[Sirenj]

I faced exactly the same problem and finally got it fixed by downgrading Java version from 9 to 8. I am using Linux Mint, PostgreSQL, migratus "1.0.2" and migratus-lein "0.5.4". The problem and the fix repeated also on my Windows laptop.

[yogthos]

Ah interesting, might be related to Java 9 modules loading. I'll take a look when I get a chance, but if you have time to investigate this a bit that would be great. :)

[yogthos]

I tracked down the problem. Migratus uses migratus.utils/find-migration-dir function to locate the migration directory. The function relies on clojure.java.classpath/classpath-directories to find the working directory. It appears that classpath-directories returns an empty sequence when running under JDK 9, and so no migration files end up being found.

[yogthos]

Looks like this is a known issue in Clojure.

[yogthos]

I pushed out a fix in version 1.0.3, @pasih could you try it and see if it works on your end.

[k2n]

I also had this issue and confirmed it worked. Thank you for your quick fix!

[yogthos]

👍