Migrate Neo4j from v4.4.23 --> v5.19.0

[ao508]

REMAINING TASKS:

• [ ] Test all migrated services in dev environment
• [ ] Make new db dump from production database
• [ ] Make db upgrade on production server
• [ ] ensure that web pages and entire SMILE workflow is working start to finish
• [ ] Confirm that all deprecation warnings are now dealt with

---

In addition to migrating the database and verifying its integrity..

Other items that will need to be done:

• create a datadump of the current database in production
• update neo4j/graphql driver and related dependencies in the dashboard codebase
• update neo4j/spring data driver and related dependencies in the smile backend codebases
• update test dependencies for smile backend codebases as well
• several end-to-end test scenarios should be done to ensure that none of the messaging workflows (specifically in the smile-server codebase) are negatively impacted by the db migration.

Coordinating migration:

• The database migration should be systematically done of course with clear communication with related parties that the services will be brought down for an extended period of time but not longer than a day.
• Document the migration steps performed on the dev server extremely well to ensure that the production server is not down for long as we should have clear steps to follow.
• ASK DIGITS FOR A SNAPSHOT PLEASE AND THANK YOU

[ao508]

Notes from #1263

PAUSED FOR THE TIME BEING

Neo4j pattern matching is supported in graphql for Neo4j v5.x.x which will require a major db version migration on our part.

It's recommended to incrementally migrate db versions. Ours is at 4.4.23 (need to double check how many migration steps this would require to reach 5.19.0)

The latest Neo4j version is v5.23.0 but APOC is supported up until v5.19.0 so far which is motivation for only migrating up to v5.19.0.

Resources and things to keep in mind:

• docker image: neo4j/neo4j-admin:5.19.0 to perform the database dump and migration
• docker image: neo4j/5.19.0 to launch neo4j with the designated version
• java21 is required for neo4j v5.19.0 (needed if running locally!)

Steps run:

1. using docker... make a database dump for the current version of the database (neo4j:4.4.23)

  neo4j-admin-dump:
    container_name: neo4j-admin-dump
    image: neo4j/neo4j-admin:4.4.23
    restart: unless-stopped
    environment:
      - NEO4J_AUTH=${NEO4J_USERNAME}/${NEO4J_PASSWORD}
      - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
      - NEO4J_dbms_allow__upgrade=true
    command: neo4j-admin dump --database=neo4j --to=/downloads/neo4j.dump
    volumes:
      - type: bind
        source: /Users/laptop/data/neo4j/data
        target: /data
      - type: bind
        source: /Users/laptop/data/neo4j/import
        target: /import
      - type: bind
        source: /Users/laptop/data/neo4j/logs
        target: /logs
      - type: bind
        source: /Users/laptop/data/neo4j/plugins
        target: /plugins
      - type: bind
        source: /Users/laptop/data/neo4j/downloads
        target: /downloads

because of the bind mounting, the data dump will be written to the mounted /downloads path

2. downloaded neo4j 5.19.0 community edition (as tar.gz, not using the neo4j desktop app)
3. from the unpacked directory:

# migrate the database
laptop@LSKI3053 neo4j-community-5.19.0 % bin/neo4j-admin database migrate neo4j

# load the database
neo4j-community-5.19.0 % bin/neo4j-admin database load neo4j --from-path=/Users/laptop/data/neo4j/downloads

4. Launch neo4j like normal via docker-compose, making sure to update the image to 5.19.0

[ao508]

PRs:

smile-dashboard: https://github.com/mskcc/smile-dashboard/pull/160/files smile-configuration: https://github.mskcc.org/cmo/smile-configuration/pull/137 smile-deployment: https://github.mskcc.org/cmo/smile-deployment/pull/185 smile-server: coming soon

[ao508]

After a lot of consideration, it might be best to migrate away from SDN+OGM hybrid to just SDN.