Migrating the Hetionet Neo4j Browser to Greene Lab infrastructure

[dhimmel]

Currently, the Neo4j Hetionet Browser is hosted on a DigitalOcean Droplet as discussed here. Unfortunately however DigitalOcean's sponsorship credits have expired and they have not returned my communications about continuing to sponsor the project. Therefore, now makes sense to transfer hosting to the Greene Lab infrastructure where hopefully we can find a more sustainable setup.

CCing @dongbohu who manages the Greene Lab servers. The Docker and instance information is available in hetnet/neo4j/docker. The instance needs not more than 20 GB disk space. Currently, we have 2 GB of RAM, but 4 GB would potentially help alleviate some of the crashes we periodically experience.

Some goals:

• consider an automated deployment pipeline of the Docker image
• see if we can monitor for downtime and restart the container

[dongbohu]

@dhimmel: I tried Neo4j Community Edition 3.5.4 on my local desktop. I had to upgrade the database files when starting the Neo4j process. So I uncommented the following line in conf/neo4j.conf:

dbms.allow_upgrade=true

and it seems to be working fine. Should we update hetionet-v1.0.db.tar.bz2 in the repo to make sure that our users are able to run the latest Neo4j?

I also noticed that the Dockerfile is based neo4j:3.2.8-enterprise: https://github.com/hetio/hetionet/blob/b7c144da0dd428def9e3c262d0bc48c050cf631d/hetnet/neo4j/docker/Dockerfile#L1 Why the enterprise version instead of community version?

[dongbohu]

About the two goals: (1) consider an automated deployment pipeline of the Docker image If certain files in this repo are changed, then the docker image should be rebuilt and pushed to docker hub, then the docker container should be restarted too. Is this what you mean?

(2) see if we can monitor for downtime and restart the container We can write a cron job (run it hourly or more frequent) to detect whether the container is down. If it is, restart the container automatically.

[dhimmel]

I had to upgrade the database files when starting the Neo4j process.

See the configuration we use in hetnet/neo4j/docker/files/neo4j.conf. We likely want to stick with the current Neo4j distro and configuration we use as much as possible... unless you are looking for extra debugging work haha.

Should we update hetionet-v1.0.db.tar.bz2 in the repo to make sure that our users are able to run the latest Neo4j?

I don't see a need to do this unless the upgrade stops working.

Why the enterprise version instead of community version?

Switched to enterprise in https://github.com/hetio/hetionet/commit/4dfa938e740ee94ddc5f6ed9de21744bb355902c to accomplish configuration options that should be supported by community but were not. For example, "query logging" and the browser.remote_content_hostname_whitelist which for some releases has been enterprise only.

Note that Neo4j enterprise is no longer open source. Therefore, in the future if we want to upgrade neo4j versions, we will switch back to community. However, the differences are minimal for the setup we have here and it's likely some of the configuration options will have changed anyways.

If certain files in this repo are changed, then the docker image should be rebuilt and pushed to docker hub, then the docker container should be restarted too. Is this what you mean?

Yes, it would be great to have CI build and deploy the docker, while ensuring docker tags remain protected: see https://github.com/docker/hub-feedback/issues/958. Let's chat about how we want to do this.

We can write a cron job (run it hourly or more frequent) to detect whether the container is down. If it is, restart the container automatically.

Yes, that sounds good. We'll have to look back at old issues as guidance like https://github.com/hetio/hetionet/issues/14, https://github.com/hetio/hetionet/issues/5, https://github.com/hetio/hetionet/issues/9, https://github.com/hetio/hetionet/issues/10.

We also should consider whether the Docker images can be used locally and whether we want to put SSL certificate logic in the container (with a possible optional Let's Encrypt SSL retrieval).

[dhimmel]

@dongbohu and I have migrated https://neo4j.het.io over to an AWS instance under the Greene Lab account. The elastic IP address for the instance is 3.212.243.105. I updated the A record with the domain registrar.