Redo-migrations

[jkhartshorne]

Should speed up database migrations, not get stuck on transactions_db.

[jkhartshorne]

Addresses https://github.com/pushkin-consortium/pushkin/issues/233

[jkhartshorne]

Can you two try out prep using this new version a few times. It should not hang on setting up transactions db. It should also be a bit faster than the old version.

[jessestorbeck]

I am getting this error every time I run prep:

jessestorbeck@Jesses-MacBook-Air-4 test % node ../pushkin-cli/build/index.js prep
Removing any .DS_Store files, if present. /Users/jessestorbeck/Desktop/test/experiments /Users/jessestorbeck/Desktop/test/pushkin package manager: yarn resetting experiments.js Started prepping API for testExp Started loading API controller for jspsych7exp Using build-if-changed for jspsych7exp_api Installing dependencies for /Users/jessestorbeck/Desktop/test/experiments/testExp/api controllers Started prepping web page for testExp Started loading web page for testExp Using build-if-changed for testExp_web Installing dependencies for /Users/jessestorbeck/Desktop/test/experiments/testExp/web page loaded docker-compose.dev.yml and pushkin.yaml. Building worker for testExp Building docker image for testexp_worker docker build /Users/jessestorbeck/Desktop/test/experiments/testExp/worker -t testexp_worker --load Spooling up databases. userMigsDir: /Users/jessestorbeck/Desktop/test/users/migrations userDatabase: localtestdb Loading migrations for testExp Updated users/config.yaml to be compatible with default site template v2+. expConfig: {"experimentName":"testExp","shortName":"testExp","apiControllers":{"mountPath":"jspsych7exp","location":"api controllers","name":"mycontroller"},"worker":{"location":"worker","service":{"image":"jspsych7exp","links":["message-queue","test_db","test_transaction_db"],"environment":{"AMQP_ADDRESS":"amqp://message-queue:5672","DB_USER":"postgres","DB_PASS":"example","DB_URL":"test_db","DB_NAME":"test_db"}}},"webPage":{"location":"web page"},"migrations":{"location":"migrations"},"seeds":{"location":""},"database":"localtestdb","logo":"logo512.png","text":"Enter your experiment description here.","tagline":"Be a citizen scientist! Try this quiz.","duration":"","productionDB":"Main"} Building jspsych7exp_api from /Users/jessestorbeck/Desktop/test/experiments/testExp/api controllers Building testExp_web from /Users/jessestorbeck/Desktop/test/experiments/testExp/web page node:internal/errors:862 const err = new Error(message); ^

Error: Command failed: docker build /Users/jessestorbeck/Desktop/test/experiments/testExp/worker -t testexp_worker --load unknown flag: --load See 'docker build --help'.

at ChildProcess.exithandler (node:child_process:419:12)
at ChildProcess.emit (node:events:511:28)
at maybeClose (node:internal/child_process:1098:16)
at ChildProcess._handle.onexit (node:internal/child_process:304:5) {

code: 125, killed: false, signal: null, cmd: 'docker build /Users/jessestorbeck/Desktop/test/experiments/testExp/worker -t testexp_worker --load', stdout: '', stderr: "unknown flag: --load\nSee 'docker build --help'.\n" }

Node.js v20.2.0

[ellissc]

Ran prep a few times without any issues (example output below), but the site had a network error after running start and checking localhost.

(base) wkw@Eugene-Onegin pushkin-cli-redo-migrations % node /Users/wkw/pushkin-dev/pushkin-cli-redo-migrations/build/index.js prep             
Removing any .DS_Store files, if present.
/Users/wkw/pushkin-dev/pushkin-cli-redo-migrations/experiments
/Users/wkw/pushkin-dev/pushkin-cli-redo-migrations/pushkin
package manager:  yarn
resetting experiments.js
Started prepping API for test
Started loading API controller for jspsych7exp
Using build-if-changed for  jspsych7exp_api
Installing dependencies for /Users/wkw/pushkin-dev/pushkin-cli-redo-migrations/experiments/test/api controllers
Started prepping web page for test
Started loading web page for test
Using build-if-changed for  test_web
Installing dependencies for /Users/wkw/pushkin-dev/pushkin-cli-redo-migrations/experiments/test/web page
loaded docker-compose.dev.yml and pushkin.yaml.
Building worker for test
Building docker image for test_worker
docker build /Users/wkw/pushkin-dev/pushkin-cli-redo-migrations/experiments/test/worker -t test_worker --load
Spooling up databases.
userMigsDir: /Users/wkw/pushkin-dev/pushkin-cli-redo-migrations/users/migrations
userDatabase: localtestdb
Loading migrations for test
Updated users/config.yaml to be compatible with default site template v2+.
expConfig:
 {"experimentName":"test","shortName":"test","apiControllers":{"mountPath":"jspsych7exp","location":"api controllers","name":"mycontroller"},"worker":{"location":"worker","service":{"image":"jspsych7exp","links":["message-queue","test_db","test_transaction_db"],"environment":{"AMQP_ADDRESS":"amqp://message-queue:5672","DB_USER":"postgres","DB_PASS":"example","DB_URL":"test_db","DB_NAME":"test_db"}}},"webPage":{"location":"web page"},"migrations":{"location":"migrations"},"seeds":{"location":""},"database":"localtestdb","logo":"logo512.png","text":"Enter your experiment description here.","tagline":"Be a citizen scientist! Try this quiz.","duration":"","productionDB":"Main"}
Building test_web from /Users/wkw/pushkin-dev/pushkin-cli-redo-migrations/experiments/test/web page
Building jspsych7exp_api from /Users/wkw/pushkin-dev/pushkin-cli-redo-migrations/experiments/test/api controllers
jspsych7exp_api is built
test_web is built
jspsych7exp_api is published locally via yalc
test_web is published locally via yalc
Loaded web page for test (test_web)
Added test to experiments.js
Writing out experiments.js
Installing combined API
Installed combined API
Installing combined front-end
Installed combined front-end
Waiting for test transaction db...
Waiting for test db...
Waiting for test db...
Waiting for test transaction db...
Waiting for test transaction db...
Waiting for test db...
Waiting for test transaction db...
Waiting for test db...
Waiting for test transaction db...
Waiting for test db...
Test transaction db is healthy
Starting migrations for test transactions DB.
Running migrations for localtransactiondb
Test test db is healthy
Running migrations for localtestdb
Ran migrations for localtransactiondb
Ran migrations for localtestdb
Running seeds on /Users/wkw/pushkin-dev/pushkin-cli-redo-migrations/experiments/test
Finished running all migrations. Shutting down database containers.
Finished building all experiment workers
updating docker-compose.dev.yml
Writing out front-end config
Building API
Building server

[jkhartshorne]

Try upgrading Docker Desktop and Docker Engine and then try again.

[jessestorbeck]

@jkhartshorne, I've tested this now after updating Docker (Docker Desktop 4.22.0, Docker Engine 24.0.5), and I can now view my site on localhost and run a basic experiment. @ellissc, if you try this, I would recommend restarting your computer after updating Docker, since I was getting some unfamiliar errors before that. After restarting, though, things behaved pretty normally. I'm not sure if prep was any faster for me, but it didn't get stuck on transactions_db (although that hasn't happened very often for me anyway).

I did have the familiar problem (pushkin-consortium/pushkin#225), where I couldn't run the experiment the first time after running start; however, my normal solution of running stop, prep, and start (in this case: node ../pushkin-cli/build/index.js stop; node ../pushkin-cli/build/index.js prep; node ../pushkin-cli/build/index.js start) still solved the issue for me, and I was able to run the experiment after that.