search

DataFabricRus / scylla-rdf

An RDF store based on ScyllaDB and Eclipse RDF4J
MIT License
8 stars 1 forks source link

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed #8

Closed so-dewy closed 5 years ago

so-dewy commented 5 years ago

I successfully deployed scylla-rdf to docker then got into RDF4J Workbench and can create repositories. The problem is that I can't do anything else, when I try to access Namespaces, Contexts, and all other tabs I get similar exceptions: 

HTTP Status 500 – Internal Server Error
Type Exception Report

Message org.eclipse.rdf4j.repository.RepositoryException: org.eclipse.rdf4j.repository.RepositoryException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /0.0.4.87:9042 (com.datastax.driver.core.exceptions.TransportException: [/0.0.4.87:9042] Cannot connect))

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

javax.servlet.ServletException: org.eclipse.rdf4j.repository.RepositoryException: org.eclipse.rdf4j.repository.RepositoryException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /0.0.4.87:9042 (com.datastax.driver.core.exceptions.TransportException: [/0.0.4.87:9042] Cannot connect))
    org.eclipse.rdf4j.workbench.proxy.WorkbenchServlet.handleRequest(WorkbenchServlet.java:140)
    org.eclipse.rdf4j.workbench.proxy.WorkbenchServlet.service(WorkbenchServlet.java:103)
    org.eclipse.rdf4j.workbench.proxy.WorkbenchGateway.service(WorkbenchGateway.java:114)
    org.eclipse.rdf4j.workbench.base.AbstractServlet.service(AbstractServlet.java:125)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
    org.eclipse.rdf4j.workbench.proxy.CacheFilter.doFilter(CacheFilter.java:61)
    org.eclipse.rdf4j.workbench.proxy.CookieCacheControlFilter.doFilter(CookieCacheControlFilter.java:53)
Root Cause

org.eclipse.rdf4j.repository.RepositoryException: org.eclipse.rdf4j.repository.RepositoryException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /0.0.4.87:9042 (com.datastax.driver.core.exceptions.TransportException: [/0.0.4.87:9042] Cannot connect))
    org.eclipse.rdf4j.http.client.SPARQLProtocolSession.execute(SPARQLProtocolSession.java:1019)
    org.eclipse.rdf4j.http.client.SPARQLProtocolSession.executeOK(SPARQLProtocolSession.java:958)
    org.eclipse.rdf4j.http.client.SPARQLProtocolSession.sendTupleQueryViaHttp(SPARQLProtocolSession.java:755)
    org.eclipse.rdf4j.http.client.SPARQLProtocolSession.getTupleQueryResult(SPARQLProtocolSession.java:690)
    org.eclipse.rdf4j.http.client.RDF4JProtocolSession.getNamespaces(RDF4JProtocolSession.java:359)
    org.eclipse.rdf4j.http.client.RDF4JProtocolSession.getNamespaces(RDF4JProtocolSession.java:344)
    org.eclipse.rdf4j.repository.http.HTTPRepositoryConnection.getNamespaces(HTTPRepositoryConnection.java:726)
    org.eclipse.rdf4j.workbench.commands.NamespacesServlet.service(NamespacesServlet.java:49)
    org.eclipse.rdf4j.workbench.base.TransformationServlet.service(TransformationServlet.java:111)
    org.eclipse.rdf4j.workbench.base.TransformationServlet.service(TransformationServlet.java:95)
    org.eclipse.rdf4j.workbench.base.AbstractServlet.service(AbstractServlet.java:125)
    org.eclipse.rdf4j.workbench.proxy.ProxyRepositoryServlet.service(ProxyRepositoryServlet.java:99)
    org.eclipse.rdf4j.workbench.proxy.WorkbenchServlet.service(WorkbenchServlet.java:188)
    org.eclipse.rdf4j.workbench.proxy.WorkbenchServlet.handleRequest(WorkbenchServlet.java:128)
    org.eclipse.rdf4j.workbench.proxy.WorkbenchServlet.service(WorkbenchServlet.java:103)
    org.eclipse.rdf4j.workbench.proxy.WorkbenchGateway.service(WorkbenchGateway.java:114)
    org.eclipse.rdf4j.workbench.base.AbstractServlet.service(AbstractServlet.java:125)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
    org.eclipse.rdf4j.workbench.proxy.CacheFilter.doFilter(CacheFilter.java:61)
    org.eclipse.rdf4j.workbench.proxy.CookieCacheControlFilter.doFilter(CookieCacheControlFilter.java:53)

Here is what I do to deploy:

  1. Build ScyllaDB Java Driver 3.7.1-scylla branch and RDF4J (rdf4j, rdf4j-storage and rdf4j-tools) develop branch.
  2. Build and deploy scylla-rdf. 
    mvn clean install
docker-compose -f docker-compose-dev.yml up -d
  3. Access http://localhost and get to RDF4J Workbench. No password is required for some reason.
  4. Create repositories.
  5. Can't access other tabs other than Modify options, but if try to add some data it throws the same exception.

Am I doing something wrong?

This is what I see in the summary of the repository: image And all repositories: image

I get the same result on two different PCs running Windows 10 Pro.

Edit: I found how to login as admin but it did't solve the issue.

KMax commented 5 years ago

Hi @NascentSoul! I guess the problem is related to ScyllaDB, since I see the exception message:

org.eclipse.rdf4j.repository.RepositoryException: org.eclipse.rdf4j.repository.RepositoryException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /0.0.4.87:9042 (com.datastax.driver.core.exceptions.TransportException: [/0.0.4.87:9042] Cannot connect))

Can you attach logs from ScyllaDB? Maybe you don't have enough RAM.

You can also try to change the 5th line in docker-compose-dev.yml to:

    command: --overprovisioned 1 --developer-mode 1

maybe it'll help.

so-dewy commented 5 years ago

Seems like not enough memory: 

> docker logs scylla-rdf_scylladb_1
running: (['/usr/lib/scylla/scylla_dev_mode_setup', '--developer-mode', '1'],)
running: (['/usr/lib/scylla/scylla_io_setup'],)
2019-07-09 10:40:41,615 CRIT Supervisor running as root (no user in config file)
2019-07-09 10:40:41,615 WARN Included extra file "/etc/supervisord.conf.d/scylla-server.conf" during parsing
2019-07-09 10:40:41,615 WARN Included extra file "/etc/supervisord.conf.d/scylla-jmx.conf" during parsing
2019-07-09 10:40:41,615 WARN Included extra file "/etc/supervisord.conf.d/scylla-housekeeping.conf" during parsing
2019-07-09 10:40:41,695 INFO RPC interface 'supervisor' initialized
2019-07-09 10:40:41,696 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2019-07-09 10:40:41,696 INFO supervisord started with pid 10
2019-07-09 10:40:42,699 INFO spawned: 'scylla-jmx' with pid 13
2019-07-09 10:40:42,702 INFO spawned: 'scylla' with pid 14
2019-07-09 10:40:42,705 INFO spawned: 'scylla-housekeeping' with pid 15
Could not initialize seastar: std::runtime_error (insufficient physical memory: needed 2147483648 available 500000000)
2019-07-09 10:40:43,529 INFO exited: scylla (exit status 1; not expected)
Connecting to http://localhost:10000
Starting the JMX server
2019-07-09 10:40:44,338 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-07-09 10:40:44,338 INFO success: scylla-housekeeping entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-07-09 10:40:45,341 INFO spawned: 'scylla' with pid 31
JMX is enabled to receive remote connections on port: 7199
Could not initialize seastar: std::runtime_error (insufficient physical memory: needed 2147483648 available 500000000)
2019-07-09 10:40:46,055 INFO exited: scylla (exit status 1; not expected)

With command: --overprovisioned 1 --developer-mode 1

> docker logs scylla-rdf_scylladb_1
running: (['/usr/lib/scylla/scylla_dev_mode_setup', '--developer-mode', '1'],)
running: (['/usr/lib/scylla/scylla_io_setup'],)
2019-07-09 10:49:50,845 CRIT Supervisor running as root (no user in config file)
2019-07-09 10:49:50,845 WARN Included extra file "/etc/supervisord.conf.d/scylla-server.conf" during parsing
2019-07-09 10:49:50,846 WARN Included extra file "/etc/supervisord.conf.d/scylla-jmx.conf" during parsing
2019-07-09 10:49:50,846 WARN Included extra file "/etc/supervisord.conf.d/scylla-housekeeping.conf" during parsing
2019-07-09 10:49:50,885 INFO RPC interface 'supervisor' initialized
2019-07-09 10:49:50,885 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2019-07-09 10:49:50,885 INFO supervisord started with pid 10
2019-07-09 10:49:51,887 INFO spawned: 'scylla-jmx' with pid 13
2019-07-09 10:49:51,889 INFO spawned: 'scylla' with pid 14
2019-07-09 10:49:51,891 INFO spawned: 'scylla-housekeeping' with pid 15
WARN  2019-07-09 10:49:52,412 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
Scylla version 3.0.8-0.20190627.23da53c4f starting ...
WARN  2019-07-09 10:49:52,517 [shard 0] init - Only 238 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
INFO  2019-07-09 10:49:52,616 [shard 0] init - Scylla API server listening on 127.0.0.1:10000 ...
INFO  2019-07-09 10:49:52,959 [shard 0] database - Row: max_vector_size: 32, internal_count: 5
2019-07-09 10:49:52,960 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-07-09 10:49:52,960 INFO success: scylla entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-07-09 10:49:52,960 INFO success: scylla-housekeeping entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
INFO  2019-07-09 10:49:52,961 [shard 1] database - Row: max_vector_size: 32, internal_count: 5
WARN  2019-07-09 10:49:52,961 [shard 0] init - I/O Scheduler is not properly configured! This is a non-supported setup, and performance is expected to be unpredictably bad.
 Reason found: none of --max-io-requests, --io-properties and --io-properties-file are set.
To properly configure the I/O Scheduler, run the scylla_io_setup utility shipped with Scylla.

WARN  2019-07-09 10:49:52,986 [shard 0] storage_service - Shutting down communications due to I/O errors until operator intervention
WARN  2019-07-09 10:49:52,986 [shard 0] storage_service - Disk error: std::system_error (error system:22, Invalid argument)
INFO  2019-07-09 10:49:52,988 [shard 0] storage_service - Stop transport: starts
ERROR 2019-07-09 10:49:53,008 [shard 0] init - Directory '/var/lib/scylla/data' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
INFO  2019-07-09 10:49:53,008 [shard 0] storage_service - Stop transport: stop_gossiping done
INFO  2019-07-09 10:49:53,008 [shard 0] storage_service - Stop transport: shutdown rpc and cql server done
INFO  2019-07-09 10:49:53,012 [shard 0] storage_service - messaging_service stopped
ERROR 2019-07-09 10:49:53,013 [shard 0] seastar - Exiting on unhandled exception: storage_io_error (Storage I/O error: 22: Invalid argument)
INFO  2019-07-09 10:49:53,015 [shard 0] compaction_manager - Asked to stop
INFO  2019-07-09 10:49:53,015 [shard 1] compaction_manager - Asked to stop
INFO  2019-07-09 10:49:53,015 [shard 0] storage_service - Stop transport: shutdown messaging_service done
INFO  2019-07-09 10:49:53,015 [shard 0] storage_service - stream_manager stopped
INFO  2019-07-09 10:49:53,015 [shard 0] storage_service - Stop transport: shutdown stream_manager done
INFO  2019-07-09 10:49:53,021 [shard 0] storage_service - Stop transport: auth shutdown
INFO  2019-07-09 10:49:53,021 [shard 0] storage_service - Stop transport: done
2019-07-09 10:49:53,147 INFO exited: scylla (exit status 1; not expected)
Connecting to http://localhost:10000
Starting the JMX server
2019-07-09 10:49:53,684 INFO spawned: 'scylla' with pid 35
WARN  2019-07-09 10:49:54,012 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
Scylla version 3.0.8-0.20190627.23da53c4f starting ...
WARN  2019-07-09 10:49:54,019 [shard 0] init - Only 238 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
INFO  2019-07-09 10:49:54,021 [shard 0] init - Scylla API server listening on 127.0.0.1:10000 ...
INFO  2019-07-09 10:49:54,027 [shard 0] database - Row: max_vector_size: 32, internal_count: 5
INFO  2019-07-09 10:49:54,041 [shard 1] database - Row: max_vector_size: 32, internal_count: 5
WARN  2019-07-09 10:49:54,041 [shard 0] init - I/O Scheduler is not properly configured! This is a non-supported setup, and performance is expected to be unpredictably bad.
 Reason found: none of --max-io-requests, --io-properties and --io-properties-file are set.
To properly configure the I/O Scheduler, run the scylla_io_setup utility shipped with Scylla.

WARN  2019-07-09 10:49:54,043 [shard 0] storage_service - Shutting down communications due to I/O errors until operator intervention
WARN  2019-07-09 10:49:54,043 [shard 0] storage_service - Disk error: std::system_error (error system:22, Invalid argument)
INFO  2019-07-09 10:49:54,043 [shard 0] storage_service - Stop transport: starts
INFO  2019-07-09 10:49:54,043 [shard 0] storage_service - Stop transport: stop_gossiping done
INFO  2019-07-09 10:49:54,043 [shard 0] storage_service - Stop transport: shutdown rpc and cql server done
INFO  2019-07-09 10:49:54,043 [shard 0] storage_service - messaging_service stopped
INFO  2019-07-09 10:49:54,043 [shard 0] storage_service - Stop transport: shutdown messaging_service done
INFO  2019-07-09 10:49:54,043 [shard 0] storage_service - stream_manager stopped
INFO  2019-07-09 10:49:54,043 [shard 0] storage_service - Stop transport: shutdown stream_manager done
INFO  2019-07-09 10:49:54,043 [shard 0] storage_service - Stop transport: auth shutdown
INFO  2019-07-09 10:49:54,043 [shard 0] storage_service - Stop transport: done
ERROR 2019-07-09 10:49:54,043 [shard 0] init - Directory '/var/lib/scylla/data' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
ERROR 2019-07-09 10:49:54,044 [shard 0] seastar - Exiting on unhandled exception: storage_io_error (Storage I/O error: 22: Invalid argument)
INFO  2019-07-09 10:49:54,044 [shard 0] compaction_manager - Asked to stop
INFO  2019-07-09 10:49:54,044 [shard 1] compaction_manager - Asked to stop
2019-07-09 10:49:54,247 INFO exited: scylla (exit status 1; not expected)
JMX is enabled to receive remote connections on port: 7199
2019-07-09 10:49:55,775 INFO spawned: 'scylla' with pid 43
WARN  2019-07-09 10:49:56,197 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
Scylla version 3.0.8-0.20190627.23da53c4f starting ...
WARN  2019-07-09 10:49:56,209 [shard 0] init - Only 238 MiB per shard; this is below the recommended minimum of 1 GiB/shard; continuing since running in developer mode
INFO  2019-07-09 10:49:56,215 [shard 0] init - Scylla API server listening on 127.0.0.1:10000 ...
INFO  2019-07-09 10:49:56,220 [shard 0] database - Row: max_vector_size: 32, internal_count: 5
INFO  2019-07-09 10:49:56,255 [shard 1] database - Row: max_vector_size: 32, internal_count: 5
WARN  2019-07-09 10:49:56,255 [shard 0] init - I/O Scheduler is not properly configured! This is a non-supported setup, and performance is expected to be unpredictably bad.
 Reason found: none of --max-io-requests, --io-properties and --io-properties-file are set.
To properly configure the I/O Scheduler, run the scylla_io_setup utility shipped with Scylla.

WARN  2019-07-09 10:49:56,260 [shard 0] storage_service - Shutting down communications due to I/O errors until operator intervention
WARN  2019-07-09 10:49:56,260 [shard 0] storage_service - Disk error: std::system_error (error system:22, Invalid argument)
INFO  2019-07-09 10:49:56,260 [shard 0] storage_service - Stop transport: starts
INFO  2019-07-09 10:49:56,260 [shard 0] storage_service - Stop transport: stop_gossiping done
INFO  2019-07-09 10:49:56,260 [shard 0] storage_service - Stop transport: shutdown rpc and cql server done
INFO  2019-07-09 10:49:56,260 [shard 0] storage_service - messaging_service stopped
INFO  2019-07-09 10:49:56,260 [shard 0] storage_service - Stop transport: shutdown messaging_service done
INFO  2019-07-09 10:49:56,260 [shard 0] storage_service - stream_manager stopped
INFO  2019-07-09 10:49:56,260 [shard 0] storage_service - Stop transport: shutdown stream_manager done
INFO  2019-07-09 10:49:56,260 [shard 0] storage_service - Stop transport: auth shutdown
INFO  2019-07-09 10:49:56,260 [shard 0] storage_service - Stop transport: done
ERROR 2019-07-09 10:49:56,260 [shard 0] init - Directory '/var/lib/scylla/data' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
ERROR 2019-07-09 10:49:56,260 [shard 0] seastar - Exiting on unhandled exception: storage_io_error (Storage I/O error: 22: Invalid argument)
INFO  2019-07-09 10:49:56,261 [shard 0] compaction_manager - Asked to stop
INFO  2019-07-09 10:49:56,261 [shard 1] compaction_manager - Asked to stop
2019-07-09 10:49:56,366 INFO exited: scylla (exit status 1; not expected)
KMax commented 5 years ago

I've updated docker-compose-dev.yml to limit the max heap size used by Scylla-RDF. You also can comment out Elasticsearch, if you don't need the full-text search for now.

so-dewy commented 5 years ago

Sorry, I forgot to emphasize that it still throws error in the logs with command: --overprovisioned 1 --developer-mode 1

ERROR 2019-07-09 10:49:53,008 [shard 0] init - Directory '/var/lib/scylla/data' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
KMax commented 5 years ago

I guess it could be related to the directory permissions on the host. Try to comment out lines 12-13, 28-29, 40-41 in docker-compose-dev.yml.

KMax commented 5 years ago

I've just updated the docker-compose.dev.yml (see c9c31d7d90ec7ebf728fd4fdad3b3a06946359e1). Try to use the new version.

so-dewy commented 5 years ago

Thank you, scylladb starts now with these changes. But it continued throwing the same exception even though scylladb worked judging by logs so I figured out that I was creating repositories wrong... In Scylla Hosts field I was entering IP of the scylladb node of different container. Feeling kinda dumb for that. Seems like everything works now.

KMax commented 5 years ago

@NascentSoul Also instead of IP addresses, you can use the names of the containers as hostnames, e.g. scylladb and elasticsearch.