Closed jaeddy closed 5 years ago
I've gotten stuck on updating docker-compose.yml
— I can't figure out how to get the Python Flask service to connect to the Neo4j server in the separate container. I've added a step to mount authentication config as well as something (wait-for-it
) to make sure the app doesn't try to connect until the neo4j server is running. However, I'm still getting an error that the connection is refused (see below).
@Pawel-Madej, @lukasz-rakoczy, @tschaffter - any ideas?
prov-service$ docker-compose up
Starting neo4j ... done
Recreating prov-server ... done
Attaching to neo4j, prov-server
neo4j | Active database: graph.db
prov-server | wait-for-it.sh: waiting 20 seconds for 127.0.0.1:7687
neo4j | Directories in use:
neo4j | home: /var/lib/neo4j
neo4j | config: /var/lib/neo4j/conf
neo4j | logs: /logs
neo4j | plugins: /var/lib/neo4j/plugins
neo4j | import: /var/lib/neo4j/import
neo4j | data: /var/lib/neo4j/data
neo4j | certificates: /var/lib/neo4j/certificates
neo4j | run: /var/lib/neo4j/run
neo4j | Starting Neo4j.
neo4j | 2019-07-15 22:02:29.670+0000 WARN Unknown config option: causal_clustering.discovery_listen_address
neo4j | 2019-07-15 22:02:29.676+0000 WARN Unknown config option: causal_clustering.raft_advertised_address
neo4j | 2019-07-15 22:02:29.677+0000 WARN Unknown config option: causal_clustering.raft_listen_address
neo4j | 2019-07-15 22:02:29.677+0000 WARN Unknown config option: ha.host.coordination
neo4j | 2019-07-15 22:02:29.678+0000 WARN Unknown config option: causal_clustering.transaction_advertised_address
neo4j | 2019-07-15 22:02:29.678+0000 WARN Unknown config option: causal_clustering.discovery_advertised_address
neo4j | 2019-07-15 22:02:29.679+0000 WARN Unknown config option: ha.host.data
neo4j | 2019-07-15 22:02:29.679+0000 WARN Unknown config option: causal_clustering.transaction_listen_address
neo4j | 2019-07-15 22:02:29.706+0000 INFO ======== Neo4j 3.5.7 ========
neo4j | 2019-07-15 22:02:29.730+0000 INFO Starting...
neo4j | 2019-07-15 22:02:33.982+0000 INFO Bolt enabled on 0.0.0.0:7687.
neo4j | 2019-07-15 22:02:37.586+0000 INFO Started.
neo4j | 2019-07-15 22:02:39.797+0000 INFO Remote interface available at http://localhost:7474/
prov-server | ./wait-for-it.sh: line 62: 9 Terminated timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT
prov-server | wait-for-it.sh: timeout occurred after waiting 20 seconds for 127.0.0.1:7687
prov-server | INFO:synprov.config:Connecting to neo4j database
prov-server | INFO:synprov:Checking connection at bolt://127.0.0.1:7687
prov-server | Traceback (most recent call last):
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 835, in _connect
prov-server | s.connect(resolved_address)
prov-server | ConnectionRefusedError: [Errno 111] Connection refused
prov-server |
prov-server | During handling of the above exception, another exception occurred:
prov-server |
prov-server | Traceback (most recent call last):
prov-server | File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
prov-server | "__main__", mod_spec)
prov-server | File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
prov-server | exec(code, run_globals)
prov-server | File "/usr/src/app/synprov/__main__.py", line 31, in <module>
prov-server | main()
prov-server | File "/usr/src/app/synprov/__main__.py", line 26, in main
prov-server | init_db()
prov-server | File "/usr/src/app/synprov/__main__.py", line 17, in init_db
prov-server | neo4j_available()
prov-server | File "/usr/src/app/synprov/__init__.py", line 18, in neo4j_available
prov-server | neo4j_connection.run('MATCH () RETURN 1 LIMIT 1')
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 533, in run
prov-server | return self.begin(autocommit=True).run(cypher, parameters, **kwparameters)
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 828, in run
prov-server | entities=entities))
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/internal/connectors.py", line 292, in run
prov-server | return self._run_1(statement, parameters, graph, keys, entities)
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/internal/connectors.py", line 241, in _run_1
prov-server | cx = self.pool.acquire()
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 719, in acquire
prov-server | return self.acquire_direct(self.address)
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 612, in acquire_direct
prov-server | connection = self.connector(address, error_handler=self.connection_error_handler)
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/internal/connectors.py", line 227, in connector
prov-server | encrypted=cx_data["secure"], **kwargs)
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 976, in connect
prov-server | raise last_error
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 966, in connect
prov-server | s = _connect(resolved_address, **config)
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 847, in _connect
prov-server | raise ServiceUnavailable("Failed to establish connection to {!r} (reason {})".format(resolved_address, error))
prov-server | neobolt.exceptions.ServiceUnavailable: Failed to establish connection to ('127.0.0.1', 7687) (reason [Errno 111] Connection refused)
prov-server exited with code 1
@jaeddy The URL that the service prov-server
must use to contact the service neo4j-dev
is https://<name of the docker service>/api/...
. Docker will automatically resolve <name of the docker service>
. So if you name the service neo4j-dev
in docker-compose.yml
, then the provenance server must connect to it with http://neo4j-dev/...
.
Ah - thanks for the tip, @tschaffter. Neo4j isn't refusing the connection anymore, but now I'm getting authentication errors...
Will try to sort out later.
$ docker-compose up
Recreating neo4j ... done
Recreating prov-server ... done
Attaching to neo4j, prov-server
prov-server | wait-for-it.sh: waiting 20 seconds for neo4j-dev:7687
neo4j | Changed password for user 'neo4j'.
neo4j | Active database: graph.db
neo4j | Directories in use:
neo4j | home: /var/lib/neo4j
neo4j | config: /var/lib/neo4j/conf
neo4j | logs: /logs
neo4j | plugins: /var/lib/neo4j/plugins
neo4j | import: /var/lib/neo4j/import
neo4j | data: /var/lib/neo4j/data
neo4j | certificates: /var/lib/neo4j/certificates
neo4j | run: /var/lib/neo4j/run
neo4j | Starting Neo4j.
neo4j | 2019-07-16 00:35:12.904+0000 WARN Unknown config option: causal_clustering.discovery_listen_address
neo4j | 2019-07-16 00:35:12.910+0000 WARN Unknown config option: causal_clustering.raft_advertised_address
neo4j | 2019-07-16 00:35:12.910+0000 WARN Unknown config option: causal_clustering.raft_listen_address
neo4j | 2019-07-16 00:35:12.910+0000 WARN Unknown config option: ha.host.coordination
neo4j | 2019-07-16 00:35:12.911+0000 WARN Unknown config option: causal_clustering.transaction_advertised_address
neo4j | 2019-07-16 00:35:12.911+0000 WARN Unknown config option: causal_clustering.discovery_advertised_address
neo4j | 2019-07-16 00:35:12.911+0000 WARN Unknown config option: ha.host.data
neo4j | 2019-07-16 00:35:12.912+0000 WARN Unknown config option: causal_clustering.transaction_listen_address
neo4j | 2019-07-16 00:35:12.938+0000 INFO ======== Neo4j 3.5.7 ========
neo4j | 2019-07-16 00:35:12.963+0000 INFO Starting...
neo4j | 2019-07-16 00:35:17.345+0000 INFO Bolt enabled on 0.0.0.0:7687.
prov-server | wait-for-it.sh: neo4j-dev:7687 is available after 9 seconds
prov-server | INFO:synprov.config:Connecting to neo4j database
neo4j | 2019-07-16 00:35:19.493+0000 WARN The client is unauthorized due to authentication failure.
prov-server | Traceback (most recent call last):
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 828, in run
prov-server | entities=entities))
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/internal/connectors.py", line 292, in run
prov-server | return self._run_1(statement, parameters, graph, keys, entities)
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/internal/connectors.py", line 241, in _run_1
prov-server | cx = self.pool.acquire()
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 719, in acquire
prov-server | return self.acquire_direct(self.address)
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 612, in acquire_direct
prov-server | connection = self.connector(address, error_handler=self.connection_error_handler)
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/internal/connectors.py", line 227, in connector
prov-server | encrypted=cx_data["secure"], **kwargs)
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 976, in connect
prov-server | raise last_error
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 968, in connect
prov-server | connection = _handshake(s, address, der_encoded_server_certificate, **config)
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 936, in _handshake
prov-server | connection.hello()
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 249, in hello
prov-server | self.sync()
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 531, in sync
prov-server | detail_delta, summary_delta = self.fetch()
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 422, in fetch
prov-server | return self._fetch()
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 464, in _fetch
prov-server | response.on_failure(summary_metadata or {})
prov-server | File "/usr/local/lib/python3.7/site-packages/neobolt/direct.py", line 778, in on_failure
prov-server | raise AuthError(message)
prov-server | neobolt.exceptions.AuthError: The client is unauthorized due to authentication failure.
prov-server |
prov-server | During handling of the above exception, another exception occurred:
prov-server |
prov-server | Traceback (most recent call last):
prov-server | File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
prov-server | "__main__", mod_spec)
prov-server | File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
prov-server | exec(code, run_globals)
prov-server | File "/usr/src/app/synprov/__main__.py", line 30, in <module>
prov-server | main()
prov-server | File "/usr/src/app/synprov/__main__.py", line 25, in main
prov-server | init_db()
prov-server | File "/usr/src/app/synprov/__main__.py", line 17, in init_db
prov-server | graph.delete_all()
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 382, in delete_all
prov-server | self.run("MATCH (a) DETACH DELETE a")
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 533, in run
prov-server | return self.begin(autocommit=True).run(cypher, parameters, **kwparameters)
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 830, in run
prov-server | raise GraphError.hydrate({"code": error.code, "message": error.message})
prov-server | File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 676, in hydrate
prov-server | _, classification, category, title = code.split(".")
prov-server | AttributeError: 'NoneType' object has no attribute 'split'
prov-server | ERROR:neobolt:Failed to write data to connection ('neo4j-dev', 7687) (Address(host='172.18.0.2', port=7687)); ("9; 'Bad file descriptor'")
prov-server exited with code 1
neo4j | 2019-07-16 00:35:20.927+0000 INFO Started.
neo4j | 2019-07-16 00:35:23.187+0000 INFO Remote interface available at http://localhost:7474/
Hi @jaeddy
Is this still an issue? I tried to start the latest setup from the master branch and the neo4j connection seems to be fine:
neo4j | 2019-07-16 12:35:06.357+0000 INFO ======== Neo4j 3.5.7 ========
neo4j | 2019-07-16 12:35:06.368+0000 INFO Starting...
neo4j | 2019-07-16 12:35:09.019+0000 INFO Bolt enabled on 0.0.0.0:7687.
prov-server | wait-for-it.sh: neo4j-dev:7687 is available after 5 seconds
prov-server | INFO:synprov.config:Connecting to neo4j database
neo4j | 2019-07-16 12:35:11.871+0000 INFO Started.
prov-server | INFO:__main__:Populating mock graph
prov-server | INFO:synprov.mock.main:Generating table of Activities...
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_1' of class 'Memoization'
neo4j | 2019-07-16 12:35:14.101+0000 INFO Remote interface available at http://localhost:7474/
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_2' of class 'Report generation'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_3' of class 'Mention'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_4' of class 'Starred'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_5' of class 'Starred'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_6' of class 'Mention'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_7' of class 'Mention'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_8' of class 'Mention'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_9' of class 'Mention'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_10' of class 'Memoization'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_11' of class 'Report generation'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_12' of class 'Mention'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_13' of class 'Starred'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_14' of class 'Tool session'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_15' of class 'Tool session'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_16' of class 'Mention'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_17' of class 'Report generation'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_18' of class 'Report generation'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_19' of class 'Memoization'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_20' of class 'Memoization'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_21' of class 'Starred'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_22' of class 'Mention'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_23' of class 'Report generation'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_24' of class 'Report generation'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_25' of class 'Starred'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_26' of class 'Starred'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_27' of class 'Memoization'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_28' of class 'Starred'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_29' of class 'Mention'
prov-server | INFO:synprov.mock.main:Building Activity 'Activity_30' of class 'Memoization'
prov-server | * Serving Flask app "synprov.config" (lazy loading)
prov-server | * Environment: production
prov-server | WARNING: This is a development server. Do not use it in a production deployment.
prov-server | Use a production WSGI server instead.
prov-server | * Debug mode: off
prov-server | INFO:werkzeug: * Running on http://localhost:8080/ (Press CTRL+C to quit)
@lukasz-rakoczy - I could never get the password-change via NEO4J_AUTH
environment variable to work on my machine.. I eventually figured out how to get the auth credentials mounted via a volume — which is more consistent with how the Travis build is handled.
Had to do some additional troubleshooting with hosts and ports to access the service outside the container, but eventually figured it out. Should be working now (additional instructions in the README).
Hi @jaeddy
I can confirm that the docker compose as it is now works fine (neo4j connection is established and the prov-service can be accessed).
However the volume solution seems to add some complexity to the deployment process so I will try to make it using env. variables. When it is ready I will create a PR so you can check if it is fine.
Hi @jaeddy
I've been working on neo4j authentication so we don't need to use volumes for that. It seems I still have no permissions to create branches in the repo but the docker-compose.yml below seems to be working to me.
What you need to do is to set the NEO4J_USERNAME and NEO4J_PASSWORD env. variables on the host system and adjust volumes so neo4j data and logs can be permanently stored.
I'm not sure if we have this requirement for the pilot but Collaboration Portal needs to talk to MongoDB via SSL secured connection. Do you plan implement secure connection for Prov Service and neo4j as well?
version: '3.2'
services:
neo4j-dev:
image: neo4j:latest
container_name: neo4j
ports:
- "7687:7687"
- "7474:7474"
volumes:
- /tmp/neo4j/data:/var/lib/neo4j/data
- /tmp/neo4j/logs:/logs
environment:
- NEO4J_AUTH=${NEO4J_USERNAME}/${NEO4J_PASSWORD}
prov-server:
build:
context: .
dockerfile: Dockerfile
image: prov-server
command: python3 -m synprov
container_name: prov-server
ports:
- "8080:8080"
environment:
- NEO4J_SCHEME=bolt
- NEO4J_HOST=neo4j-dev
- NEO4J_PORT=7687
- NEO4J_USERNAME=${NEO4J_USERNAME}
- NEO4J_PASSWORD=${NEO4J_PASSWORD}
- FLASK_HOST=0.0.0.0
links:
- neo4j-dev
Thanks, @lukasz-rakoczy! I'll check out the latest docker-compose configuration.
I think i sent you an invite to collaborate on the repo a while ago, but it probably got buried in your inbox. I just sent another — let me know if you get it. That should give you write permission (ability to create and push to new branches).
We do need to enable SSL for the provenance service for the pilot phase (see #22). I'm hoping to get some help on that issue and have asked @Pawel-Madej to take a look. Let me (or Pawel) know if you can help out as well.
These haven't been touched since the switch to neo4j. Need to get working to enable deployment and testing.