Open hobson opened 6 years ago
schlevik
commented
6 years ago No, you have to load the kb first.
I.E.:
curl -XPOST "http://localhost:8917/_kb/dbpedia-2016/_load"
That'll download the DBpedia dump (approx. 30 3GB iirc)
For a small subset use the obama KB.
hobson
commented
6 years ago Pretty sure I did that. The output shows DBpedial loaded. Perhaps the download wasn't complete?
--Hobson
On Mon, Dec 18, 2017 at 6:32 AM, Viktor Schlegel notifications@github.com wrote:
No, you have to load the kb first. I.E.: curl -XPOST "http://localhost:8917/_kb/dbpedia-2016/_load" That'll download the DBpedia dump (approx. 30GB iirc) For a small subset use the obama KB.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Lambda-3/Stargraph/issues/27#issuecomment-352441629, or mute the thread https://github.com/notifications/unsubscribe-auth/AAR39wX_DM4KJAr28K--1nar97MdkLsdks5tBndggaJpZM4REeEt .
schlevik
commented
6 years ago Your output
elastic | [2017-12-16T21:41:09,491][INFO ][o.e.g.GatewayService ] [ejB6sWq] recovered [0] indices into cluster_statehints that the ES indices (used for semantic approximation) are not created (nor populated). In order to do so you have to load the knowledge base. This process downloads the knowledge base and creates the indices after that. As it takes some time it is entirely possible, that the process was not completed. At the moment, there is no implementation to check the indexing progress over the REST interface, so you have to verify that all data was loaded and indexed by checking the log for a
"Successful: [dbpedia-2016.entities ...message. Please note that
curl -XGET http://localhost:8917/_kbonly lists the knowledge bases which are preconfigured to use, it does not indicate whether the knowledge bases were actually initialized (i.e. downloaded and indexed).
I suggest you try to initialize the knowledge bases again and then check the log output for suspicious activity (such as Fail to load, failing: or some exceptions)
Hope I could help you, regards
hobson
commented
6 years ago Awesome! Trying now.
--Hobson
On Wed, Dec 20, 2017 at 7:29 AM, Viktor Schlegel notifications@github.com wrote:
Your output
elastic | [2017-12-16T21:41:09,491][INFO ][o.e.g.GatewayService ] [ejB6sWq] recovered [0] indices into cluster_state
hints that the ES indices (used for semantic approximation) are not created (nor populated). In order to do so you have to load the knowledge base. This process downloads the knowledge base and creates the indices after that. As it takes some time it is entirely possible, that the process was not completed. At the moment, there is no implementation to check the indexing progress over the REST interface, so you have to verify that all data was loaded and indexed by checking the log for a
"Successful: [dbpedia-2016.entities ...
message. Please note that
curl -XGET http://localhost:8917/_kb
only lists the knowledge bases which are preconfigured to use, it does not indicate whether the knowledge bases were actually initialized (i.e. downloaded and indexed).
I suggest you try to initialize the knowledge bases again and then check the log output for suspicious activity (such as Fail to load, failing: or some exceptions)
Hope I could help you, regards
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Lambda-3/Stargraph/issues/27#issuecomment-353094097, or mute the thread https://github.com/notifications/unsubscribe-auth/AAR3989j3aEIxWSp8oG9mZwV3we09jeUks5tCSfagaJpZM4REeEt .
hobson
commented
6 years ago Unfortunately, the DB seems to have failed to load, without any indication as to why. The download seems to go fine, loading into the DB does not.
elastic | [2017-12-21T00:03:02,753][INFO ][o.e.n.Node ] initialized
elastic | [2017-12-21T00:03:02,753][INFO ][o.e.n.Node ] [ejB6sWq] starting ...
elastic | [2017-12-21T00:03:03,122][INFO ][o.e.t.TransportService ] [ejB6sWq] publish_address {172.18.0.2:9300}, bound_addresses {0.0.0.0:9300}
elastic | [2017-12-21T00:03:03,141][INFO ][o.e.b.BootstrapCheck ] [ejB6sWq] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elastic | [2017-12-21T00:03:06,389][INFO ][o.e.c.s.ClusterService ] [ejB6sWq] new_master {ejB6sWq}{ejB6sWq4QyGhK1nj1ba7AQ}{xg_KZf3QTq6YnJX5U9FvXQ}{172.18.0.2}{172.18.0.2:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
elastic | [2017-12-21T00:03:06,430][INFO ][o.e.h.HttpServer ] [ejB6sWq] publish_address {172.18.0.2:9200}, bound_addresses {0.0.0.0:9200}
elastic | [2017-12-21T00:03:06,431][INFO ][o.e.n.Node ] [ejB6sWq] started
elastic | [2017-12-21T00:03:06,448][INFO ][o.e.g.GatewayService ] [ejB6sWq] recovered [0] indices into cluster_state
stargraphv2 | 2017-12-21 00:05:25,001 WARN [core] Loading ALL DATA of 'dbpedia-2016'. This can take some time ;) ..
stargraphv2 | 2017-12-21 00:05:25,005 INFO [index] Loading dbpedia-2016.relations, [reset=true, limit=-1]
stargraphv2 | 2017-12-21 00:05:25,015 WARN [index] Old data will be DELETED!
stargraphv2 | 2017-12-21 00:05:25,025 INFO [index] Awaiting Loader finalization..
stargraphv2 | 2017-12-21 00:05:25,059 INFO [core] Downloading from: 'http://data.lambda3.org/dbpedia_en-2016.hdt'
stargraphv2 | 2017-12-21 00:20:31,935 INFO [core] Loading '/usr/share/stargraph/data/dbpedia-2016/facts/triples.hdt', useIndex=false
elastic exited with code 137
stargraphv2 | 2017-12-21 00:21:39,427 INFO [core] Main Namespaces: [dbr]
stargraphv2 | 2017-12-21 00:21:39,436 INFO [core] Namespace resource: default-namespace.txt
stargraphv2 | 2017-12-21 00:21:39,476 INFO [core] Creating dbpedia-2016.relations data provider
stargraphv2 | 2017-12-21 00:21:39,864 INFO [core] processors = ProcessorChain{chain=[length-filter, stop-property-filter, sink-duplicate, wordnet]}
stargraphv2 | 2017-12-21 00:21:39,878 ERROR [index] Loader failure.
stargraphv2 | net.stargraph.core.index.IndexingException: Indexing Error
stargraphv2 | at net.stargraph.core.impl.elastic.ElasticIndexer.beforeLoad(ElasticIndexer.java:82)
stargraphv2 | at net.stargraph.core.index.BaseIndexer.doBeforeLoad(BaseIndexer.java:154)
stargraphv2 | at net.stargraph.core.index.BaseIndexer.lambda$doLoad$0(BaseIndexer.java:215)
stargraphv2 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
stargraphv2 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
stargraphv2 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
stargraphv2 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
stargraphv2 | at java.lang.Thread.run(Thread.java:745)
stargraphv2 | Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{4t3h2d7GSay6LjMuFWAfBg}{elastic}{172.18.0.2:9300}]
stargraphv2 | at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:328)
stargraphv2 | at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:226)
stargraphv2 | at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
stargraphv2 | at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:345)
stargraphv2 | at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
stargraphv2 | at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
stargraphv2 | at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
stargraphv2 | at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
stargraphv2 | at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:62)
stargraphv2 | at net.stargraph.core.impl.elastic.ElasticIndexer.beforeLoad(ElasticIndexer.java:68)
stargraphv2 | ... 7 common frames omitted
stargraphv2 | 2017-12-21 00:21:39,884 INFO [index] Loader is finishing..
stargraphv2 | 2017-12-21 00:21:39,892 WARN [index] Nothing was loaded!
stargraphv2 | 2017-12-21 00:21:39,893 INFO [index] Loader is done.
stargraphv2 | 2017-12-21 00:21:39,900 ERROR [core] Fail to load dbpedia-2016.relations
stargraphv2 | 2017-12-21 00:21:39,910 INFO [index] Loading dbpedia-2016.facts, [reset=true, limit=-1]
stargraphv2 | 2017-12-21 00:21:39,912 WARN [index] Old data will be DELETED!
stargraphv2 | 2017-12-21 00:21:39,914 INFO [index] Awaiting Loader finalization..
stargraphv2 | 2017-12-21 00:21:39,947 INFO [core] Main Namespaces: [dbr]
stargraphv2 | 2017-12-21 00:21:39,962 INFO [core] Namespace resource: default-namespace.txt
stargraphv2 | 2017-12-21 00:21:39,978 INFO [core] Creating dbpedia-2016.facts data provider
stargraphv2 | 2017-12-21 00:21:39,993 INFO [core] processors = ProcessorChain{chain=[length-filter, entity-classifier, regex-filter]}
stargraphv2 | 2017-12-21 00:21:39,997 ERROR [index] Loader failure.
stargraphv2 | net.stargraph.core.index.IndexingException: Indexing Error
stargraphv2 | at net.stargraph.core.impl.elastic.ElasticIndexer.beforeLoad(ElasticIndexer.java:82)
stargraphv2 | at net.stargraph.core.index.BaseIndexer.doBeforeLoad(BaseIndexer.java:154)
stargraphv2 | at net.stargraph.core.index.BaseIndexer.lambda$doLoad$0(BaseIndexer.java:215)
stargraphv2 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
stargraphv2 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
stargraphv2 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
stargraphv2 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
stargraphv2 | at java.lang.Thread.run(Thread.java:745)
stargraphv2 | Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{rjiSzc7oQHCs_A4xVK0K-g}{elastic}{172.18.0.2:9300}]
stargraphv2 | at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:328)
stargraphv2 | at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:226)
stargraphv2 | at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
stargraphv2 | at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:345)
stargraphv2 | at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
stargraphv2 | at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
stargraphv2 | at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
stargraphv2 | at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
stargraphv2 | at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:62)
stargraphv2 | at net.stargraph.core.impl.elastic.ElasticIndexer.beforeLoad(ElasticIndexer.java:68)
stargraphv2 | ... 7 common frames omitted
stargraphv2 | 2017-12-21 00:21:40,001 INFO [index] Loader is finishing..
stargraphv2 | 2017-12-21 00:21:40,002 WARN [index] Nothing was loaded!
stargraphv2 | 2017-12-21 00:21:40,004 INFO [index] Loader is done.
stargraphv2 | 2017-12-21 00:21:40,006 ERROR [core] Fail to load dbpedia-2016.facts
stargraphv2 | 2017-12-21 00:21:40,008 INFO [index] Loading dbpedia-2016.entities, [reset=true, limit=-1]
stargraphv2 | 2017-12-21 00:21:40,011 WARN [index] Old data will be DELETED!
stargraphv2 | 2017-12-21 00:21:40,013 INFO [index] Awaiting Loader finalization..
stargraphv2 | 2017-12-21 00:21:40,017 INFO [core] Main Namespaces: [dbr]
stargraphv2 | 2017-12-21 00:21:40,018 INFO [core] Namespace resource: default-namespace.txt
stargraphv2 | 2017-12-21 00:21:40,095 INFO [core] Creating dbpedia-2016.entities data provider
stargraphv2 | 2017-12-21 00:21:40,481 INFO [core] processors = ProcessorChain{chain=[length-filter, sink-duplicate]}
stargraphv2 | 2017-12-21 00:21:40,484 ERROR [index] Loader failure.
stargraphv2 | net.stargraph.core.index.IndexingException: Indexing Error
stargraphv2 | at net.stargraph.core.impl.elastic.ElasticIndexer.beforeLoad(ElasticIndexer.java:82)
stargraphv2 | at net.stargraph.core.index.BaseIndexer.doBeforeLoad(BaseIndexer.java:154)
stargraphv2 | at net.stargraph.core.index.BaseIndexer.lambda$doLoad$0(BaseIndexer.java:215)
stargraphv2 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
stargraphv2 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
stargraphv2 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
stargraphv2 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
stargraphv2 | at java.lang.Thread.run(Thread.java:745)
stargraphv2 | Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{ke_kjhNySAezH7s638SNQQ}{elastic}{172.18.0.2:9300}]
stargraphv2 | at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:328)
stargraphv2 | at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:226)
stargraphv2 | at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
stargraphv2 | at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:345)
stargraphv2 | at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
stargraphv2 | at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
stargraphv2 | at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
stargraphv2 | at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
stargraphv2 | at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:62)
stargraphv2 | at net.stargraph.core.impl.elastic.ElasticIndexer.beforeLoad(ElasticIndexer.java:68)
stargraphv2 | ... 7 common frames omitted
stargraphv2 | 2017-12-21 00:21:40,488 INFO [index] Loader is finishing..
stargraphv2 | 2017-12-21 00:21:40,489 WARN [index] Nothing was loaded!
stargraphv2 | 2017-12-21 00:21:40,490 INFO [index] Loader is done.
stargraphv2 | 2017-12-21 00:21:40,491 ERROR [core] Fail to load dbpedia-2016.entities
stargraphv2 | 2017-12-21 00:21:40,492 WARN [core] Sucessful: [], failing: [dbpedia-2016.relations, dbpedia-2016.facts, dbpedia-2016.entities] stargraphv2 | Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{4t3h2d7GSay6LjMuFWAfBg}{elastic}{172.18.0.2:9300}]aah yes.
Try this docker-config.yml
version: '2'
services:
elastic:
image: elasticsearch:5.4.0
container_name: elastic
command: "-Ecluster.name=StarGraphV2Cluster -Ebootstrap.memory_lock=true -Ediscovery.zen.minimum_master_nodes=1 -Ecluster.routing.allocation.disk.threshold_enabled=false -Etransport.host=0.0.0.0 -Ediscovery.type=single-node"
expose:
- 9300
volumes:
- ./esdata:/usr/share/elasticsearch/data
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 8g
cap_add:
- IPC_LOCK
environment:
- "ES_JAVA_OPTS=-Xms7g -Xmx7g"
stargraph:
image: lambdacube/stargraph:latest
container_name: stargraphv2
ports:
- 8917:8917
volumes:
- ./logs:/usr/share/stargraph/logs
- ./data:/usr/share/stargraph/data
mem_limit: 8g
environment:
- "STARGRAPH_JAVA_OPTS=-Xms7g -Xmx7g"note the
-Etransport.host=0.0.0.0 -Ediscovery.type=single-nodeappendix.
I couldn't get it to run with the provided docker-compose.yml either - for elastic 5.1 it would crash on startup due to some bug, additionally you have to allow elastic to listen to "0.0.0.0" in order to be able to access it from outside the elastic docker container, which would fail the bootstrap check for me until I added -Ediscovery.type=single-node which is only available from elastic 5.4 on.
Greetings
I can't get the README instructions for spinning up a docker container to work.
docker for Mac installed
On macOS High Sierra Version 10.13.1
2.8 GHz and 8 i7 cores (4 dedicated to the docker machine) 16GB memory (8 dedicated to docker-machine)
In a Max OSX Terminal
All the required data/indices seem to be loaded:
And the server container seems to boot up without crashing...
In the client terminal (same Mac OSX host)
But queries don't work