search

dhimmel / integrate

Scripts and resources to create Hetionet v1.0, a heterogeneous network for drug repurposing
https://doi.org/10.15363/thinklab.4
31 stars 16 forks source link

Connection refused when using hetio/py2neo to export to neo4j #11

Closed veleritas closed 7 years ago

veleritas commented 7 years ago

Hi Daniel,

Do you remember what you needed to do to increase the number of concurrent open files allowable by neo4j? At the moment I get a warning that says:

WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual.
Starting Neo4j Server...WARNING: not changing user
process [6305]... waiting for server to be ready..... OK.
http://localhost:7500/ is ready.

Also, I think they way that neo4j is started with a subprocess call in Python is giving me issues, since I trying to start neo4j 3.1.1 in the same manner causes the service to shut down immediately before any Python code can be executed.

dhimmel commented 7 years ago

So the warning is because your operating system is set to allow at most 1024 open files handles. However, it's just a warning... not an error. IIRC, the number of open files in Neo4j is only large if you have many users modifying the database at a given time. I don't think this should happen for our use cases... but if it does there will be an error message in the log.

It looks like in Neo4j 3.1, it's now easier to change the open file limit. But I suspect this isn't your problem.

I trying to start neo4j 3.1.1 in the same manner causes the service to shut down immediately before any Python code can be executed.

If we can get a log here that would be great!

veleritas commented 7 years ago

The strange thing is that the neo4j.log file is empty when this happens. It seems that the server is starting, refusing a connection to the Python code, and then shuts down immediately.

neo4j-import.ipynb outputs something like:

Starting Neo4j.
WARNING: Max 1024 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
Started neo4j (pid 2085). By default, it is available at http://localhost:7474/
There may be a short delay until the server is ready.
See /home/ubuntu/fold0/integrate/neo4j/neo4j-community-3.1.1_rephetio-v2.0/logs/neo4j.log for current status.
Neo4j is running at pid 2085
Stopping Neo4j.. stopped
neo4j/neo4j-community-3.1.1_rephetio-v2.0 Connection refused

The code looks like:

def create_instance(version, db_id, port=7474, overwrite=False):
    """Create neo4j instance"""

    # Download neo4j
    filename = '{}-unix.tar.gz'.format(version)
    path = os.path.join('neo4j', filename)
    if not os.path.exists(path):
        url = 'http://neo4j.com/artifact.php?name={}'.format(filename)
        urllib.request.urlretrieve(url, path)

    # Extract to file
    tar_file = tarfile.open(path, 'r:gz')
    tar_file.extractall('neo4j')
    directory = os.path.join('neo4j', '{}_{}'.format(version, db_id))

    if os.path.isdir(directory) and overwrite:
        shutil.rmtree(directory)
    os.rename(os.path.join('neo4j', version), directory)

    # modify configuration settings
    path = os.path.join(directory, 'conf', 'neo4j.conf')

    replace_text(path,
        '#dbms.security.auth_enabled=false',
        'dbms.security.auth_enabled=false'
    )

    replace_text(path,
        "#dbms.connectors.default_advertised_address=localhost",
        "dbms.connectors.default_advertised_address=localhost"
    )

    replace_text(path,
        "#dbms.connector.http.listen_address=:7474",
        "dbms.connector.http.listen_address=:{}".format(port)
    )

    replace_text(path,
        "dbms.connector.https.enabled=true",
        "dbms.connector.https.enabled=false"
    )

    replace_text(path,
        "dbms.connector.bolt.enabled=true",
        "dbms.connector.bolt.enabled=false"
    )

    return directory

def hetnet_to_neo4j(path, neo4j_dir, port, database_path='data/graph.db'):
    """
    Read a hetnet from file and import it into a new neo4j instance.
    """
    neo4j_bin = os.path.join(neo4j_dir, 'bin', 'neo4j')

    subprocess.run([neo4j_bin, 'start'])

    subprocess.run([neo4j_bin, 'status'])

    error = None
    try:
        graph = hetio.readwrite.read_graph(path)       
        uri = 'http://localhost:{}/db/data/'.format(port)
        hetio.neo4j.export_neo4j(graph, uri, 1000, 250)
    except Exception as e:
        error = e
        print(neo4j_dir, e)
    finally:
        subprocess.run([neo4j_bin, 'stop'])

#     if not error:
#         database_dir = os.path.join(neo4j_dir, database_path)
#         remove_logs(database_dir)

def remove_logs(database_dir):
    """Should only run when server is shutdown."""
    filenames = os.listdir(database_dir)
    removed = list()
    for filename in filenames:
        if (filename.startswith('neostore.transaction.db') or
            filename.startswith('messages.log')):
            path = os.path.join(database_dir, filename)
            os.remove(path)
            removed.append(filename)
    return removed

def main():

    with open("../crossval_idx.txt", "r") as fin:
        crossval_idx = int(fin.read().strip())

    neo4j_version = 'neo4j-community-3.1.1'
    db_name = 'rephetio-v2.0'

    port_0 = 8500

    neo4j_dir = create_instance(neo4j_version, db_name, port_0, overwrite=True)

    hetnet_to_neo4j('data/hetnet.json.bz2', neo4j_dir, port_0)

if __name__ == "__main__":
    main()
dhimmel commented 7 years ago

My guess is that the following python line is actually shutting down the server:

    finally:
        subprocess.run([neo4j_bin, 'stop'])

This is happing after the error of Connection refused occurs. I'm guessing this has to do the URI being incorrect. Can you print the URI that is used in:

        uri = 'http://localhost:{}/db/data/'.format(port)
        hetio.neo4j.export_neo4j(graph, uri, 1000, 250)

Then see what that does in your browser. You can comment out finally to keep the server running for debugging (but then you may have to manually kill the neo4j processes in the terminal, e.g. killall neo4j).

veleritas commented 7 years ago

It's running on AWS, so I'll have to try some things to see if I can access the neo4j instance. It seems that the refused connection is occurring even if I comment out the stop command in the finally block.

dhimmel commented 7 years ago

Are you able to get this file from AWS: /home/ubuntu/fold0/integrate/neo4j/neo4j-community-3.1.1_rephetio-v2.0/logs/neo4j.log? That hopefully will have a message for why the server is stopping.

veleritas commented 7 years ago

Here are the logs:

neo4j.log:

2017-01-24 23:53:59.165+0000 INFO  Starting...
2017-01-24 23:54:00.450+0000 INFO  Bolt enabled on localhost:7687.
2017-01-24 23:54:02.822+0000 INFO  Started.
2017-01-24 23:54:03.856+0000 INFO  Remote interface available at http://localhost:7474/

and debug.log:

2017-01-24 23:53:59.256+0000 WARN  [o.n.i.p.PageCache] The dbms.memory.pagecache.size setting has not been configured. It is recommended that this setting is always explicitly configured, to ensure the system has a balanced configuration. Until then, a computed heuristic value of 1553045504 bytes will be used instead. 
2017-01-24 23:53:59.320+0000 INFO  [o.n.k.i.f.CommunityEditionModule] No locking implementation specified, defaulting to 'community'
2017-01-24 23:53:59.415+0000 INFO  [o.n.k.AvailabilityGuard] Requirement makes database unavailable: Database available
2017-01-24 23:53:59.434+0000 INFO  [o.n.k.i.DiagnosticsManager] --- INITIALIZED diagnostics START ---
2017-01-24 23:53:59.434+0000 INFO  [o.n.k.i.DiagnosticsManager] Neo4j Kernel properties:
2017-01-24 23:53:59.434+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.enabled=true
2017-01-24 23:53:59.434+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.security.auth_enabled=false
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.windows_service_name=neo4j
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.directories.neo4j_home=/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.directories.import=import
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.edition=community
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.enabled=true
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.https.enabled=false
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.jvm.additional=-Dunsupported.dbms.udc.source=tarball
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] Diagnostics providers:
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] org.neo4j.kernel.configuration.Config
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] org.neo4j.kernel.info.DiagnosticsManager
2017-01-24 23:53:59.435+0000 INFO  [o.n.k.i.DiagnosticsManager] SYSTEM_MEMORY
2017-01-24 23:53:59.436+0000 INFO  [o.n.k.i.DiagnosticsManager] JAVA_MEMORY
2017-01-24 23:53:59.436+0000 INFO  [o.n.k.i.DiagnosticsManager] OPERATING_SYSTEM
2017-01-24 23:53:59.436+0000 INFO  [o.n.k.i.DiagnosticsManager] JAVA_VIRTUAL_MACHINE
2017-01-24 23:53:59.436+0000 INFO  [o.n.k.i.DiagnosticsManager] CLASSPATH
2017-01-24 23:53:59.436+0000 INFO  [o.n.k.i.DiagnosticsManager] LIBRARY_PATH
2017-01-24 23:53:59.436+0000 INFO  [o.n.k.i.DiagnosticsManager] SYSTEM_PROPERTIES
2017-01-24 23:53:59.436+0000 INFO  [o.n.k.i.DiagnosticsManager] LINUX_SCHEDULERS
2017-01-24 23:53:59.436+0000 INFO  [o.n.k.i.DiagnosticsManager] NETWORK
2017-01-24 23:53:59.436+0000 INFO  [o.n.k.i.DiagnosticsManager] System memory information:
2017-01-24 23:53:59.438+0000 INFO  [o.n.k.i.DiagnosticsManager] Total Physical memory: 3.86 GB
2017-01-24 23:53:59.439+0000 INFO  [o.n.k.i.DiagnosticsManager] Free Physical memory: 3.14 GB
2017-01-24 23:53:59.439+0000 INFO  [o.n.k.i.DiagnosticsManager] Committed virtual memory: 3.31 GB
2017-01-24 23:53:59.439+0000 INFO  [o.n.k.i.DiagnosticsManager] Total swap space: 0.00 B
2017-01-24 23:53:59.439+0000 INFO  [o.n.k.i.DiagnosticsManager] Free swap space: 0.00 B
2017-01-24 23:53:59.439+0000 INFO  [o.n.k.i.DiagnosticsManager] JVM memory information:
2017-01-24 23:53:59.439+0000 INFO  [o.n.k.i.DiagnosticsManager] Free  memory: 34.27 MB
2017-01-24 23:53:59.440+0000 INFO  [o.n.k.i.DiagnosticsManager] Total memory: 62.00 MB
2017-01-24 23:53:59.440+0000 INFO  [o.n.k.i.DiagnosticsManager] Max   memory: 988.00 MB
2017-01-24 23:53:59.446+0000 INFO  [o.n.k.i.DiagnosticsManager] Garbage Collector: G1 Young Generation: [G1 Eden Space, G1 Survivor Space]
2017-01-24 23:53:59.446+0000 INFO  [o.n.k.i.DiagnosticsManager] Garbage Collector: G1 Old Generation: [G1 Eden Space, G1 Survivor Space, G1 Old Gen]
2017-01-24 23:53:59.447+0000 INFO  [o.n.k.i.DiagnosticsManager] Memory Pool: Code Cache (Non-heap memory): committed=3.69 MB, used=3.67 MB, max=240.00 MB, threshold=0.00 B
2017-01-24 23:53:59.447+0000 INFO  [o.n.k.i.DiagnosticsManager] Memory Pool: Metaspace (Non-heap memory): committed=11.25 MB, used=10.58 MB, max=-1.00 B, threshold=0.00 B
2017-01-24 23:53:59.448+0000 INFO  [o.n.k.i.DiagnosticsManager] Memory Pool: Compressed Class Space (Non-heap memory): committed=1.50 MB, used=1.34 MB, max=1.00 GB, threshold=0.00 B
2017-01-24 23:53:59.448+0000 INFO  [o.n.k.i.DiagnosticsManager] Memory Pool: G1 Eden Space (Heap memory): committed=37.00 MB, used=17.00 MB, max=-1.00 B, threshold=?
2017-01-24 23:53:59.448+0000 INFO  [o.n.k.i.DiagnosticsManager] Memory Pool: G1 Survivor Space (Heap memory): committed=2.00 MB, used=2.00 MB, max=-1.00 B, threshold=?
2017-01-24 23:53:59.448+0000 INFO  [o.n.k.i.DiagnosticsManager] Memory Pool: G1 Old Gen (Heap memory): committed=23.00 MB, used=8.22 MB, max=988.00 MB, threshold=0.00 B
2017-01-24 23:53:59.448+0000 INFO  [o.n.k.i.DiagnosticsManager] Operating system information:
2017-01-24 23:53:59.449+0000 INFO  [o.n.k.i.DiagnosticsManager] Operating System: Linux; version: 4.4.0-53-generic; arch: amd64; cpus: 2
2017-01-24 23:53:59.449+0000 INFO  [o.n.k.i.DiagnosticsManager] Max number of file descriptors: 65536
2017-01-24 23:53:59.449+0000 INFO  [o.n.k.i.DiagnosticsManager] Number of open file descriptors: 100
2017-01-24 23:53:59.542+0000 INFO  [o.n.k.i.DiagnosticsManager] Process id: 2032@localhost
2017-01-24 23:53:59.542+0000 INFO  [o.n.k.i.DiagnosticsManager] Byte order: LITTLE_ENDIAN
2017-01-24 23:53:59.542+0000 INFO  [o.n.k.i.DiagnosticsManager] Local timezone: Etc/UTC
2017-01-24 23:53:59.542+0000 INFO  [o.n.k.i.DiagnosticsManager] JVM information:
2017-01-24 23:53:59.542+0000 INFO  [o.n.k.i.DiagnosticsManager] VM Name: OpenJDK 64-Bit Server VM
2017-01-24 23:53:59.542+0000 INFO  [o.n.k.i.DiagnosticsManager] VM Vendor: Oracle Corporation
2017-01-24 23:53:59.542+0000 INFO  [o.n.k.i.DiagnosticsManager] VM Version: 25.111-b14
2017-01-24 23:53:59.542+0000 INFO  [o.n.k.i.DiagnosticsManager] JIT compiler: HotSpot 64-Bit Tiered Compilers
2017-01-24 23:53:59.543+0000 INFO  [o.n.k.i.DiagnosticsManager] VM Arguments: [-XX:+UseG1GC, -XX:-OmitStackTraceInFastThrow, -XX:+AlwaysPreTouch, -XX:+UnlockExperimentalVMOptions, -XX:+TrustFinalNonStaticFields, -XX:+DisableExplicitGC, -Djdk.tls.ephemeralDHKeySize=2048, -Dunsupported.dbms.udc.source=tarball, -Dfile.encoding=UTF-8]
2017-01-24 23:53:59.543+0000 INFO  [o.n.k.i.DiagnosticsManager] Java classpath:
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-import-tool-3.1.1.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-shell-3.1.1.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jackson-mapper-asl-1.9.13.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-browser-2.0.0-M10.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/nashorn.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/lucene-core-5.5.0.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jetty-xml-9.2.9.v20150224.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/commons-io-2.4.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/lucene-analyzers-common-5.5.0.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/lucene-queryparser-5.5.0.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-cypher-compiler-2.3-2.3.8.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/asm-5.0.4.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jackson-core-asl-1.9.13.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-resource-3.1.1.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-lucene-index-3.1.1.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-kernel-3.1.1.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jackson-jaxrs-1.9.13.jar
2017-01-24 23:53:59.545+0000 INFO  [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-logging-3.1.1.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/localedata.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-bolt-3.1.1.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jetty-http-9.2.9.v20150224.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-csv-3.1.1.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-lucene-upgrade-3.1.1.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/commons-beanutils-1.8.3.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/cldrdata.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-8-openjdk-amd64/jre/classes
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-udc-3.1.1.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/bcprov-jdk15on-1.53.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-cypher-frontend-2.3-2.3.8.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/javax.servlet-api-3.1.0.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/mimepull-1.9.3.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jetty-server-9.2.9.v20150224.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/scala-library-2.11.8.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-dbms-3.1.1.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/parboiled-scala_2.11-1.1.7.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-graph-algo-3.1.1.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/lucene-codecs-5.5.0.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jersey-servlet-1.19.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jetty-servlet-9.2.9.v20150224.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-security-3.1.1.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/scala-reflect-2.11.8.jar
2017-01-24 23:53:59.546+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jetty-util-9.2.9.v20150224.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jersey-server-1.19.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/caffeine-2.3.3.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-jmx-3.1.1.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jetty-security-9.2.9.v20150224.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-command-line-3.1.1.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/commons-digester-2.1.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/icedtea-sound.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jsr311-api-1.1.2.r612.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/sunjce_provider.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jersey-multipart-1.19.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/parboiled-core-1.1.7.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/concurrentlinkedhashmap-lru-1.4.2.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/commons-logging-1.1.1.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-common-3.1.1.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jetty-webapp-9.2.9.v20150224.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-codegen-3.1.1.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jline-2.12.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-server-3.1.1.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-io-3.1.1.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/bcpkix-jdk15on-1.53.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/jaccess.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/sunpkcs11.jar
2017-01-24 23:53:59.547+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/commons-lang3-3.3.2.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/commons-lang-2.6.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-graphdb-api-3.1.1.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/sunec.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-cypher-frontend-3.0-3.0.8.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/rhino-1.7R4.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-graph-matching-3.1.1.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/conf/
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/server-api-3.1.1.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/zipfs.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-unsafe-3.1.1.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-cypher-3.1.1.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-cypher-compiler-3.1-3.1.1.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/commons-compress-1.12.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/opencsv-2.3.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-collections-3.1.1.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-cypher-frontend-3.1-3.1.1.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-primitive-collections-3.1.1.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jersey-core-1.19.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-consistency-check-3.1.1.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/plugins/*
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/dnsns.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/netty-all-4.1.6.Final.jar
2017-01-24 23:53:59.548+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/jetty-io-9.2.9.v20150224.jar
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/plugins/
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/commons-configuration-1.10.jar
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/lib/neo4j-cypher-compiler-3.0-3.0.8.jar
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager] Library path:
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager] /usr/java/packages/lib/amd64
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager] /usr/lib/x86_64-linux-gnu/jni
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager] /lib/x86_64-linux-gnu
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager] /usr/lib/x86_64-linux-gnu
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager] /usr/lib/jni
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager] /lib
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager] /usr/lib
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager] System.properties:
2017-01-24 23:53:59.549+0000 INFO  [o.n.k.i.DiagnosticsManager] sun.boot.library.path = /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] path.separator = :
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] file.encoding.pkg = sun.io
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] user.country = US
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] sun.java.launcher = SUN_STANDARD
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] sun.os.patch.level = unknown
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] user.dir = /home/ubuntu/temp
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] sun.jnu.encoding = UTF-8
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.udc.source = tarball
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] sun.management.compiler = HotSpot 64-Bit Tiered Compilers
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] user.home = /home/ubuntu
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] user.timezone = Etc/UTC
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] file.encoding = UTF-8
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] user.name = ubuntu
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] sun.java.command = org.neo4j.server.CommunityEntryPoint --home-dir=/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby --config-dir=/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/conf
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] sun.arch.data.model = 64
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] user.language = en
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] awt.toolkit = sun.awt.X11.XToolkit
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] jdk.tls.ephemeralDHKeySize = 2048
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] file.separator = /
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] sun.io.unicode.encoding = UnicodeLittle
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] sun.cpu.endian = little
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] sun.cpu.isalist = 
2017-01-24 23:53:59.550+0000 INFO  [o.n.k.i.DiagnosticsManager] Linux scheduler information:
2017-01-24 23:53:59.582+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.604+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.605+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.605+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.605+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.605+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.605+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.605+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.606+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.606+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.606+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.606+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.606+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.606+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.607+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.607+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.607+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.607+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.607+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.608+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.608+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.608+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.608+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.608+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.608+0000 INFO  [o.n.k.i.DiagnosticsManager] none
2017-01-24 23:53:59.608+0000 INFO  [o.n.k.i.DiagnosticsManager] Network information:
2017-01-24 23:53:59.611+0000 INFO  [o.n.k.i.DiagnosticsManager] Interface eth0:
2017-01-24 23:53:59.612+0000 INFO  [o.n.k.i.DiagnosticsManager]     address: fe80:0:0:0:405:16ff:fe3c:7bc0%eth0
2017-01-24 23:53:59.612+0000 INFO  [o.n.k.i.DiagnosticsManager]     address: 172.30.0.226
2017-01-24 23:53:59.612+0000 INFO  [o.n.k.i.DiagnosticsManager] Interface lo:
2017-01-24 23:53:59.612+0000 INFO  [o.n.k.i.DiagnosticsManager]     address: 0:0:0:0:0:0:0:1%lo
2017-01-24 23:53:59.612+0000 INFO  [o.n.k.i.DiagnosticsManager]     address: 127.0.0.1
2017-01-24 23:53:59.612+0000 INFO  [o.n.k.i.DiagnosticsManager] --- INITIALIZED diagnostics END ---
2017-01-24 23:53:59.655+0000 INFO  [o.n.b.v.r.WorkerFactory] No SSL certificate found, generating a self-signed certificate..
2017-01-24 23:54:00.450+0000 INFO  [o.n.b.v.r.WorkerFactory] Bolt Server extension loaded.
2017-01-24 23:54:00.450+0000 INFO  [o.n.b.v.r.WorkerFactory] Bolt enabled on localhost:7687.
2017-01-24 23:54:00.558+0000 INFO  [o.n.k.i.s.f.RecordFormatSelector] Selected format 'RecordFormat:StandardV3_0[v0.A.7]' for the new store
2017-01-24 23:54:00.866+0000 INFO  [o.n.k.a.i.l.LuceneLabelScanStore] No lucene scan store index found, this might just be first use. Preparing to rebuild.
2017-01-24 23:54:01.014+0000 INFO  [o.n.k.NeoStoreDataSource] Opened logical log [/home/ubuntu/temp/neo4j/neo4j-community-3.1.1_toby/data/databases/graph.db/neostore.transaction.db.0] version=0, lastTxId=1 (clean)
2017-01-24 23:54:01.021+0000 WARN  [o.n.k.i.s.MetaDataStore] Missing counts store, rebuilding it.
2017-01-24 23:54:01.088+0000 INFO  [o.n.k.a.i.l.LuceneLabelScanStore] Rebuilding lucene scan store, this may take a while
2017-01-24 23:54:01.097+0000 INFO  [o.n.k.a.i.l.LuceneLabelScanStore] Lucene scan store rebuilt (roughly 0 nodes)
2017-01-24 23:54:01.099+0000 INFO  [o.n.k.i.DatabaseHealth] Database health set to OK
2017-01-24 23:54:01.483+0000 INFO  [o.n.k.AvailabilityGuard] Fulfilling of requirement makes database available: Database available
2017-01-24 23:54:01.483+0000 INFO  [o.n.k.i.f.GraphDatabaseFacadeFactory] Database is now ready
2017-01-24 23:54:01.484+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for KernelDiagnostics:Versions START ---
2017-01-24 23:54:01.484+0000 INFO  [o.n.k.i.DiagnosticsManager] Graph Database: community single StoreId{creationTime=1485302040727, randomId=151943160252206529, storeVersion=15531981201765894, upgradeTime=1485302040727, upgradeId=1}
2017-01-24 23:54:01.484+0000 INFO  [o.n.k.i.DiagnosticsManager] Kernel version: 3.1.1,89b7bf20d72f757945f8c32988e4e4c1c83ce013
2017-01-24 23:54:01.484+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for KernelDiagnostics:Versions END ---
2017-01-24 23:54:01.485+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for NEO_STORE_VERSIONS START ---
2017-01-24 23:54:01.485+0000 INFO  [o.n.k.i.DiagnosticsManager] Store versions:
2017-01-24 23:54:01.485+0000 INFO  [o.n.k.i.DiagnosticsManager] Store versions:
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   ArrayPropertyStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   NodeStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   StringPropertyStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   PropertyIndexStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   StringPropertyStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   ArrayPropertyStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   PropertyStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   RelationshipStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   StringPropertyStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   RelationshipTypeStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   StringPropertyStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   LabelTokenStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   SchemaStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   RelationshipGroupStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   NeoStore v0.A.7
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for NEO_STORE_VERSIONS END ---
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for NEO_STORE_ID_USAGE START ---
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager] Id usage:
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager] Id usage:
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   ArrayPropertyStore: used=1 high=0
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   NodeStore: used=0 high=-1
2017-01-24 23:54:01.486+0000 INFO  [o.n.k.i.DiagnosticsManager]   StringPropertyStore: used=1 high=0
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   PropertyIndexStore: used=0 high=-1
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   StringPropertyStore: used=1 high=0
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   ArrayPropertyStore: used=1 high=0
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   PropertyStore: used=0 high=-1
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   RelationshipStore: used=0 high=-1
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   StringPropertyStore: used=1 high=0
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   RelationshipTypeStore: used=0 high=-1
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   StringPropertyStore: used=1 high=0
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   LabelTokenStore: used=0 high=-1
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   SchemaStore: used=1 high=0
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   RelationshipGroupStore: used=1 high=0
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager]   NeoStore: used=15 high=14
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for NEO_STORE_ID_USAGE END ---
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for NEO_STORE_RECORDS START ---
2017-01-24 23:54:01.487+0000 INFO  [o.n.k.i.DiagnosticsManager] Neostore records:
2017-01-24 23:54:01.488+0000 INFO  [o.n.k.i.DiagnosticsManager] TIME (Creation time): 1485302040727
2017-01-24 23:54:01.488+0000 INFO  [o.n.k.i.DiagnosticsManager] RANDOM_NUMBER (Random number for store id): 151943160252206529
2017-01-24 23:54:01.488+0000 INFO  [o.n.k.i.DiagnosticsManager] LOG_VERSION (Current log version): 0
2017-01-24 23:54:01.488+0000 INFO  [o.n.k.i.DiagnosticsManager] LAST_TRANSACTION_ID (Last committed transaction): 1
2017-01-24 23:54:01.488+0000 INFO  [o.n.k.i.DiagnosticsManager] STORE_VERSION (Store format version): 15531981201765894
2017-01-24 23:54:01.488+0000 INFO  [o.n.k.i.DiagnosticsManager] FIRST_GRAPH_PROPERTY (First property record containing graph properties): -1
2017-01-24 23:54:01.488+0000 INFO  [o.n.k.i.DiagnosticsManager] LAST_CONSTRAINT_TRANSACTION (Last committed transaction containing constraint changes): 0
2017-01-24 23:54:01.488+0000 INFO  [o.n.k.i.DiagnosticsManager] UPGRADE_TRANSACTION_ID (Transaction id most recent upgrade was performed at): 1
2017-01-24 23:54:01.488+0000 INFO  [o.n.k.i.DiagnosticsManager] UPGRADE_TIME (Time of last upgrade): 1485302040727
2017-01-24 23:54:01.489+0000 INFO  [o.n.k.i.DiagnosticsManager] LAST_TRANSACTION_CHECKSUM (Checksum of last committed transaction): 0
2017-01-24 23:54:01.489+0000 INFO  [o.n.k.i.DiagnosticsManager] UPGRADE_TRANSACTION_CHECKSUM (Checksum of transaction id the most recent upgrade was performed at): 0
2017-01-24 23:54:01.489+0000 INFO  [o.n.k.i.DiagnosticsManager] LAST_CLOSED_TRANSACTION_LOG_VERSION (Log version where the last transaction commit entry has been written into): 0
2017-01-24 23:54:01.489+0000 INFO  [o.n.k.i.DiagnosticsManager] LAST_CLOSED_TRANSACTION_LOG_BYTE_OFFSET (Byte offset in the log file where the last transaction commit entry has been written into): 16
2017-01-24 23:54:01.489+0000 INFO  [o.n.k.i.DiagnosticsManager] LAST_TRANSACTION_COMMIT_TIMESTAMP (Commit time timestamp for last committed transaction): 0
2017-01-24 23:54:01.489+0000 INFO  [o.n.k.i.DiagnosticsManager] UPGRADE_TRANSACTION_COMMIT_TIMESTAMP (Commit timestamp of transaction the most recent upgrade was performed at): 0
2017-01-24 23:54:01.489+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for NEO_STORE_RECORDS END ---
2017-01-24 23:54:01.489+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for TRANSACTION_RANGE START ---
2017-01-24 23:54:01.489+0000 INFO  [o.n.k.i.DiagnosticsManager] Transaction log:
2017-01-24 23:54:01.490+0000 INFO  [o.n.k.i.DiagnosticsManager] No transactions found in any log
2017-01-24 23:54:01.490+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for TRANSACTION_RANGE END ---
2017-01-24 23:54:01.490+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for KernelDiagnostics:StoreFiles START ---
2017-01-24 23:54:01.490+0000 INFO  [o.n.k.i.DiagnosticsManager] Disk space on partition (Total / Free / Free %): 12540243968 / 7446077440 / 59
Storage files: (filename : modification date - size)
2017-01-24 23:54:01.491+0000 INFO  [o.n.k.i.DiagnosticsManager]   index:
2017-01-24 23:54:01.491+0000 INFO  [o.n.k.i.DiagnosticsManager]   - Total: 2017-01-24T23:54:00+0000 - 0.00 B
2017-01-24 23:54:01.492+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore: 2017-01-24T23:54:00+0000 - 8.00 kB
2017-01-24 23:54:01.492+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.counts.db.a: 2017-01-24T23:54:01+0000 - 96.00 B
2017-01-24 23:54:01.492+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.492+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.labeltokenstore.db: 2017-01-24T23:54:00+0000 - 0.00 B
2017-01-24 23:54:01.492+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.labeltokenstore.db.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.492+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.labeltokenstore.db.names: 2017-01-24T23:54:00+0000 - 8.00 kB
2017-01-24 23:54:01.492+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.labeltokenstore.db.names.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.492+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.nodestore.db: 2017-01-24T23:54:00+0000 - 0.00 B
2017-01-24 23:54:01.492+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.nodestore.db.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.492+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.nodestore.db.labels: 2017-01-24T23:54:00+0000 - 8.00 kB
2017-01-24 23:54:01.493+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.nodestore.db.labels.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.493+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.propertystore.db: 2017-01-24T23:54:00+0000 - 0.00 B
2017-01-24 23:54:01.493+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.propertystore.db.arrays: 2017-01-24T23:54:00+0000 - 8.00 kB
2017-01-24 23:54:01.493+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.propertystore.db.arrays.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.493+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.propertystore.db.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.493+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.propertystore.db.index: 2017-01-24T23:54:00+0000 - 0.00 B
2017-01-24 23:54:01.493+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.propertystore.db.index.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.493+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.propertystore.db.index.keys: 2017-01-24T23:54:00+0000 - 8.00 kB
2017-01-24 23:54:01.493+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.propertystore.db.index.keys.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.493+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.propertystore.db.strings: 2017-01-24T23:54:00+0000 - 8.00 kB
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.propertystore.db.strings.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.relationshipgroupstore.db: 2017-01-24T23:54:00+0000 - 8.00 kB
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.relationshipgroupstore.db.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.relationshipstore.db: 2017-01-24T23:54:00+0000 - 0.00 B
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.relationshipstore.db.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.relationshiptypestore.db: 2017-01-24T23:54:00+0000 - 0.00 B
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.relationshiptypestore.db.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.relationshiptypestore.db.names: 2017-01-24T23:54:00+0000 - 8.00 kB
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.relationshiptypestore.db.names.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.schemastore.db: 2017-01-24T23:54:00+0000 - 8.00 kB
2017-01-24 23:54:01.494+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.schemastore.db.id: 2017-01-24T23:54:00+0000 - 9.00 B
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]   neostore.transaction.db.0: 2017-01-24T23:54:01+0000 - 16.00 B
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]   schema:
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]     label:
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]       lucene:
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]         labelStore:
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]           1:
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]             write.lock: 2017-01-24T23:54:00+0000 - 0.00 B
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]           - Total: 2017-01-24T23:54:00+0000 - 0.00 B
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]           failure-message: 2017-01-24T23:54:00+0000 - 16.00 kB
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]         - Total: 2017-01-24T23:54:00+0000 - 16.00 kB
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]       - Total: 2017-01-24T23:54:00+0000 - 16.00 kB
2017-01-24 23:54:01.495+0000 INFO  [o.n.k.i.DiagnosticsManager]     - Total: 2017-01-24T23:54:00+0000 - 16.00 kB
2017-01-24 23:54:01.496+0000 INFO  [o.n.k.i.DiagnosticsManager]   - Total: 2017-01-24T23:54:00+0000 - 16.00 kB
2017-01-24 23:54:01.496+0000 INFO  [o.n.k.i.DiagnosticsManager]   store_lock: 2017-01-24T23:54:00+0000 - 0.00 B
2017-01-24 23:54:01.496+0000 INFO  [o.n.k.i.DiagnosticsManager] --- STARTED diagnostics for KernelDiagnostics:StoreFiles END ---
2017-01-24 23:54:02.822+0000 INFO  [o.n.k.i.DiagnosticsManager] --- SERVER STARTED START ---
2017-01-24 23:54:03.856+0000 INFO  [o.n.k.i.DiagnosticsManager] --- SERVER STARTED END ---
veleritas commented 7 years ago

I've simplified the code a little to try and isolate the problem. Right now all it's trying to do is establish a connection to the more default neo4j 3.1.1 (port 7474). Here is the Python traceback I'm getting:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/py2neo/database/__init__.py in __new__(cls, *uris, **settings)
    317         try:
--> 318             inst = cls.__instances[key]
    319         except KeyError:

KeyError: (<class 'py2neo.database.Graph'>, <ServerAddress settings={'host': 'localhost', 'http_port': 7474}>, 'data')

During handling of the above exception, another exception occurred:

ConnectionRefusedError                    Traceback (most recent call last)
/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py in submit(method, uri, body, headers)
    321         try:
--> 322             response = send()
    323         except BadStatusLine as err:

/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py in send(reconnect)
    316             log.debug("> %s: %s", key, value)
--> 317         http.request(xstr(method), xstr(uri.absolute_path_reference), body, headers)
    318         return http.getresponse(**getresponse_args)

/home/ubuntu/anaconda3/envs/test/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1238         """Send a complete request to the server."""
-> 1239         self._send_request(method, url, body, headers, encode_chunked)
   1240 

/home/ubuntu/anaconda3/envs/test/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1284             body = _encode(body, 'body')
-> 1285         self.endheaders(body, encode_chunked=encode_chunked)
   1286 

/home/ubuntu/anaconda3/envs/test/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1233             raise CannotSendHeader()
-> 1234         self._send_output(message_body, encode_chunked=encode_chunked)
   1235 

/home/ubuntu/anaconda3/envs/test/lib/python3.6/http/client.py in _send_output(self, message_body, encode_chunked)
   1025         del self._buffer[:]
-> 1026         self.send(msg)
   1027 

/home/ubuntu/anaconda3/envs/test/lib/python3.6/http/client.py in send(self, data)
    963             if self.auto_open:
--> 964                 self.connect()
    965             else:

/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py in connect(self)
     79                                              socket_timeout,
---> 80                                              self.source_address)
     81         self.sock.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1)

/home/ubuntu/anaconda3/envs/test/lib/python3.6/socket.py in create_connection(address, timeout, source_address)
    721     if err is not None:
--> 722         raise err
    723     else:

/home/ubuntu/anaconda3/envs/test/lib/python3.6/socket.py in create_connection(address, timeout, source_address)
    712                 sock.bind(source_address)
--> 713             sock.connect(sa)
    714             return sock

ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

SocketError                               Traceback (most recent call last)
<ipython-input-5-e91dea05684c> in <module>()
      4 location = download(version, name)
      5 
----> 6 connect("hetnet.json.bz2", location)

<ipython-input-4-e87b965de116> in connect(netloc, neo4j_dir)
      9     graph = hetio.readwrite.read_graph(netloc)
     10     uri = 'http://localhost:7474/db/data'
---> 11     hetio.neo4j.export_neo4j(graph, uri, 1000, 250)
     12     subprocess.run([neo4j_bin, 'stop'])

/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/hetio/neo4j.py in export_neo4j(graph, uri, node_queue, edge_queue)
     16 def export_neo4j(graph, uri, node_queue=100, edge_queue=100):
     17     """Export hetnet to neo4j"""
---> 18     db_graph = py2neo.Graph(uri)
     19 
     20     # Delete all existing nodes

/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/py2neo/database/__init__.py in __new__(cls, *uris, **settings)
    325             use_bolt = address.bolt
    326             if use_bolt is None:
--> 327                 use_bolt = version_tuple(inst.__remote__.get().content["neo4j_version"]) >= (3,)
    328             if use_bolt:
    329                 auth = get_auth(address)

/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/py2neo/database/http.py in get(self, headers, redirect_limit, **kwargs)
    152         kwargs.update(cache=True)
    153         try:
--> 154             response = self.__base.get(headers=headers, redirect_limit=redirect_limit, **kwargs)
    155         except (ClientError, ServerError) as error:
    156             if error.status_code == UNAUTHORIZED:

/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py in get(self, if_modified_since, headers, redirect_limit, **kwargs)
    964             object from which content can be read
    965         """
--> 966         return self.__get_or_head("GET", if_modified_since, headers, redirect_limit, **kwargs)
    967 
    968     def put(self, body=None, headers=None, **kwargs):

/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py in __get_or_head(self, method, if_modified_since, headers, redirect_limit, **kwargs)
    941             headers["If-Modified-Since"] = formatdate(datetime_to_timestamp(if_modified_since), usegmt=True)
    942         rq = Request(method, self.uri, None, headers)
--> 943         return rq.submit(redirect_limit=redirect_limit, **kwargs)
    944 
    945     def head(self, if_modified_since=None, headers=None, redirect_limit=5, **kwargs):

/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py in submit(self, redirect_limit, **response_kwargs)
    431         uri = self.uri
    432         while True:
--> 433             http, rs = submit(self.method, uri, self.body, self.headers)
    434             status_class = rs.status // 100
    435             if status_class == 3:

/home/ubuntu/anaconda3/envs/test/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py in submit(method, uri, body, headers)
    360                                       host_port=uri.host_port)
    361         else:
--> 362             raise SocketError(code, description, host_port=uri.host_port)
    363     else:
    364         return http, response

SocketError: Connection refused
veleritas commented 7 years ago

It seems that the problem is that the new binary Bolt interface is the default, and the hetio package is trying to connect to that by default.

dhimmel commented 7 years ago

I think the issue is getting caused by the use of Neo4j 3.x rather than 2.x. Obviously, Neo4j 3.x is better, but it may take some work to upgrade this repo to use 3.x.

The environment.yml specifies py2neo==2.0.8. You could try upgrading to py2neo==3.1.2 (latest):

pip install --upgrade py2neo==3.1.2

That may fix the issue, but I'm worried the py2neo API may have changed in version 3. Worth a try though.

Maybe first check this answer. Could that fix this? (would be odd if that was the issue)

If neither of these fix the issue, we may need to consult the Neo4j slack of slack overflow.

dhimmel commented 7 years ago

It seems that the problem is that the new binary Bolt interface is the default, and the hetio package is trying to connect to that by default.

Ah, there may be a way to get bolt to work or to disable it.

veleritas commented 7 years ago

Yes I think I'm going to need to update your hetio package to make it work with Neo4j 3. I'm already on all the latest packages of everything.

dhimmel commented 7 years ago

So do you think the issue is in this line of hetio.neo4j.export_neo4j:

db_graph = py2neo.Graph(uri)
veleritas commented 7 years ago

Yes, in my current testing code I can get the following to work:

def connect(netloc, neo4j_dir):
    neo4j_bin = os.path.join(neo4j_dir, 'bin', 'neo4j')

    subprocess.run([neo4j_bin, 'start'])
    subprocess.run([neo4j_bin, 'status'])

#     temp = Graph("bolt://localhost:7687/db/data")

    temp = Graph(host="localhost", bolt = True)

    temp.delete_all()

    subprocess.run([neo4j_bin, 'status'])

#     graph = hetio.readwrite.read_graph(netloc)
#     uri = 'http://localhost:7474/db/data'
#     hetio.neo4j.export_neo4j(graph, uri, 1000, 250)
    subprocess.run([neo4j_bin, 'stop'])

Connecting to neo4j 3.1.1 directly through py2neo doesn't seem to give me any problems. However I can't specify it to use the new bolt protocol by giving it just a string, as seen in temp = Graph("bolt://localhost:7687/db/data").

There is a setting to turn bolt off in the configuration files, but I'm going to see if I can get things to work with bolt, and resort to the http way if all else fails.

dhimmel commented 7 years ago

Currently, https://neo4j.het.io is running Neo4j 3.1.0 and exposes the following ports: 80, 443, 7687.

py2neo==3.1.2 is struggling, but here is some of my findings:

import py2neo

# Works, but fails without the port (makes sense)
py2neo.Graph('http://neo4j.het.io:80')
# Works, but fails without the port (makes sense)
py2neo.Graph('https://neo4j.het.io:443')

# Fails with and without the port (does not make sense)
py2neo.Graph('bolt://neo4j.het.io:7687')

So I do think py2neo is a bit buggy here as py2neo.Graph('bolt://neo4j.het.io:7687') should work.

@veleritas your issue may be slightly different. The bolt connection cannot use the same port as http(s). The default bolt port is 7687, but it won't work to have multiple neo4j servers open at the same time using that port.

However, it's good that you can get temp = Graph(host="localhost", bolt = True). In the worst case scenario, we could modify hetio so you can pass it a uri or an already opened py2neo.Graph.

veleritas commented 7 years ago

Alright, I think I have figured out what's wrong. It seems like there is nothing wrong with py2neo except that it doesn't play well with neo4j sometimes.

The current situation is that if neo4j is started from the bash terminal by hand using ./neo4j start, and then py2neo tries to establish a connection, then everything works just fine. However, if neo4j is started from Python via a subprocess call, or if the same ./neo4j start command is started from a bash script, then py2neo is no longer able to establish a connection to the database.

I'm not really sure how to solve this though, since manually starting neo4j isn't really the best of options.

veleritas commented 7 years ago

Alright. It turns out that the code needs to wait for 5 seconds (not 4 or less) via a sleep command in order for the connection to work properly. Maddening.

dhimmel commented 7 years ago

Alright. It turns out that the code needs to wait for 5 seconds (not 4 or less) via a sleep command in order for the connection to work properly. Maddening.

Ahh good to know!

I still think there is a py2neo problem as described in https://github.com/technige/py2neo/issues/605. If I'm correct, you'll still have trouble connecting to Neo4j 3.1 using bolt with just at URI.

So it sounds like I don't need to edit hetio, but that a sleep timer should be implemented in neo4j-import.ipynb proceeding:

    neo4j_bin = os.path.join(neo4j_dir, 'bin', 'neo4j')
    subprocess.run([neo4j_bin, 'start'])

However, I would need to update hetio if you want to use bolt and py2neo doesn't address https://github.com/technige/py2neo/issues/605 soon.

veleritas commented 7 years ago

I think editing hetio may be inevitable, but I can help you there since I'm already in the process of making edits. Some things in py2neo have changed from version 2 to version 3, so hetio needs to be updated to keep up.

For example, see this issue.

dhimmel commented 7 years ago

I can help you there since I'm already in the process of making edits.

Awesome. If you make a pull request into hetio that's the most helpful. We could then release a v0.3.0. I see you've opened https://github.com/dhimmel/hetio/issues/4, so we can discuss there.

veleritas commented 7 years ago

It does seem like the URI interpreter in py2neo has some sort of bug at the moment. Removing the bolt:// from the URI fixes things for me. Strange...