Closed funkjo closed 2 years ago
Hi, thanks for contacting us.
ServiceUnavailable: Unable to retrieve routing information
can have several causes and usually means connectivity issues. To help you, please provide further information:
Lastly, inspecting the debug log often reveals the underlying problem. Please do so by adding this before starting the driver and post the output.
import logging
import sys
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
logging.getLogger("neo4j").addHandler(handler)
logging.getLogger("neo4j").setLevel(logging.DEBUG)
Hi @robsdedude , at our organization we are using enterprise neo4j
, i am running into similar error when i enabled logging i see the following statements being logged
[#0000] C:
ServiceUnavailable: Timed out trying to establish connection to ResolvedIPv4Address(('ab.ab.abc.abc', 7687))
I increased the connection timeout connection_timeout from default (30) to 60 https://neo4j.com/docs/api/python-driver/4.3/api.html#driver-configuration
but still encountering the same error , is there anything else which can be done to overcome this error or at-least narrow down to the actual cause Can you please share your thoughts
neo4j version : 4.2.9 (enterprise) python driver version : 4.4.0
thanks!!
This is really hard to debug for me as this is most certainly a network issue. The driver is telling you that the server did not respond in time. Make sure you server is running and reachable at the IP address you assume. Can you connect to the neo4j browser at http://ab.ab.abc.abc:7474
?
I'll close this issue as it's not driver related. Feel free to follow-up here nevertheless if you think there's something else I can help you with.
@robsdedude yes i can connect at (http://ab.ab.abc.abc:7474)
through browser and via application as well, if 10 request per second
are made to application 6 go through (connect to neo4j and performs queries), 4 fail with this error.
Server is up in running (was same response from neo4j support team as well) then how come drivers are unable to connect for the rest of the 4 (any limitation at neo4j server level)?
By default, the server can only handle (compute things for) up to 400 connections at the same time. You can change this value https://neo4j.com/docs/operations-manual/current/reference/configuration-settings/#config_dbms.connector.bolt.thread_pool_max_size . But do you have more than 400 driver-sessions executing work at the same time? Maybe you can find something helpful in the server logs. I'm just afraid I won't be of much help interpreting this, as I mostly work on the Python driver.
Hey @robsdedude, might be naive question but when you say 400 driver-sessions , you mean 400 sessions connections (at the same time) , something like below
with self.driver.session() as session
result = session.write_transaction(
self._create_and_return_friendship, person1_name, person2_name)
or
you mean 400 connections like below
self.driver = GraphDatabase.driver(uri, auth=(user, password))
is there a way at neo4j log level or server level to determine , how many connections are existing at given point of time (as you mentioned above 400 default value, want to check actual value) Can you please confirm. Thanks!!
The driver opens and keeps connections open only as long as needed. So in fact neither GraphDatabase.driver(uri, auth=(user, password))
nor driver.session()
acquires a connection. Only ongoing transactions and unconsumed result streams will keep a connection open.
Another idea: are you running a single server or a cluster? If it's a single server, you could try connecting to it with the bolt://...
scheme. This way it will not fail when trying to acquire a routing table but (if at all) when trying to execute work. This might make it a little easier to debug.
Finally, sharing the whole log (you can redact IP addresses and replace your queries with dummy queries) would really make debugging this easier for me.
@robsdedude We are running cluster. (we have 2 databases on. the server, but currently there is no transactions on 1 of the database it just has some data stored on it, and other one is actively used by the application )
ServiceUnavailable error execute_read 'cypher_query'.
query=UNWIND $json AS message MATCH (a:anonymous {anonymous_id: message.anonymous_id})-[*1..6]-(b:anonymous)
WITH b LIMIT 1099
RETURN b as identifiers
json={'anonymous_id': 'aaaaaaaa-aaaa-aaae-aaaa-aaaaaaaaaaaa'
> C: ServiceUnavailable: Timed out trying to establish connection to ResolvedIPv4Address(('ab.ab.abc.abc', 7687))
`bolt_result = session.read_transaction(self._execute_, cypher_query=cypher_query, json=payload)
File "/.ve/lib/python3.7/site-packages/neo4j/work/simple.py", line 396, in read_transaction
return self._run_transaction(READ_ACCESS, transaction_function, *args, **kwargs)
File "/.ve/lib/python3.7/site-packages/neo4j/work/simple.py", line 352, in _run_transaction
raise errors[-1]
File "/.ve/lib/python3.7/site-packages/neo4j/work/simple.py", line 322, in _run_transaction
self._open_transaction(access_mode=access_mode, metadata=metadata, timeout=timeout)
File "/.ve/lib/python3.7/site-packages/neo4j/work/simple.py", line 255, in _open_transaction
self._connect(access_mode=access_mode)
File "/.ve/lib/python3.7/site-packages/neo4j/work/simple.py", line 108, in _connect
super()._connect(access_mode)
File "/.ve/lib/python3.7/site-packages/neo4j/work/__init__.py", line 83, in _connect
bookmarks=self._bookmarks
File "/.ve/lib/python3.7/site-packages/neo4j/io/__init__.py", line 1179, in acquire
bookmarks=bookmarks
File "/.ve/lib/python3.7/site-packages/neo4j/io/__init__.py", line 1125, in ensure_routing_table_is_fresh
database_callback=database_callback
File "/.ve/lib/python3.7/site-packages/neo4j/io/__init__.py", line 1094, in update_routing_table
raise ServiceUnavailable("Unable to retrieve routing information")
neo4j.exceptions.ServiceUnavailable: Unable to retrieve routing information
For the successful one's we have logs something like below not sure if it is helpful
C: <ROUTING TABLE ENSURE FRESH> {None: RoutingTable(database=None routers={IPv4Address(('ab.ab.abc.abc', 7687))}, readers={}, writers={}, last_updated_time=130.306414024, ttl=0), 'neo4j': RoutingTable(database='neo4j' routers={IPv4Address(('ab.ab.abc.abc', 7687)), IPv4Address(('ab.ab.abc.abc', 7687)), IPv4Address(('ab.ab.abc.abc, 7687))}, readers={IPv4Address(('ab.ab.abc.abc', 7687)), IPv4Address(('ab.ab.abc.abc', 7687))}, writers={IPv4Address(('ab.ab.abc.abc', 7687))}, last_updated_time=457451.686306685, ttl=300)}
C: <ROUTING> Checking table freshness (readonly=True)
C: <ROUTING> Table expired=False
C: <ROUTING> Table routers={IPv4Address(('ab.ab.abc.abc', 7687)), IPv4Address(('ab.ab.abc.abc', 7687)), IPv4Address(('ab.ab.abc.abc', 7687))}
C: <ROUTING> Table has_server_for_mode=True
[#0000] C: <ACQUIRE ADDRESS> database='neo4j' address=IPv4Address(('ab.ab.abc.abc', 7687))
Please let me know if anything specific you're looking for.
Hey @robsdedude any insight on my above logs please ?
There is nothing in there that gives me a hint on where the problem could be. It certainly is a network issue.
bolt://'ab.ab.abc.abc:7687
(replace with the IP address of the server that failed) and inspect the log output. Maybe that helps.ServiceUnavailable: Unable to retrieve routing information
.@funkjo Did you manage to understand the source for this issue?
Have you been able to find a concrete solution for this? I am running into the same issue. Tried all the steps described in the posts about it but the issue won't go away. A response would be much appreciated.
I am able to get this working by using neo4j+ssc:// instead of neo4j+s.
I am able to get this working by using neo4j+ssc:// instead of neo4j+s.
This suggests, that you might not have the required root CA certificate installed on your client machine. Hence, the SSL library does not trust the certificate it receives from the server.
Also, note that while +ssc
gives you encryption of the traffic, it cannot protect you from man-in-the-middle attacks as it skips the certificate verification all together.
Hello. I have a issue with this module. Here is my code.
from neo4j import GraphDatabase
from utils.color import Color
import logging
import psycopg2
import json
class Connection:
# def __init__(self):
# """Initialize the database connection"""
# logging.info(f"{Color.YELLOW}Connecting to database...{Color.END}")
# def connect(self):
# """Connect to the database"""
# try:
# self.config = json.load(open("test_config.json", "r"))
# self.connection = GraphDatabase.driver(
# self.config.get("gdb").get("uri"),
# auth=(
# self.config.get("gdb").get("user"),
# self.config.get("gdb").get("password")
# )
# )
# # self.connection = psycopg2.connect(
# # host=self.config.get("db").get("host"),
# # database=self.config.get("db").get("database"),
# # user=self.config.get("db").get("user"),
# # password=self.config.get("db").get("password"),
# # )
# logging.info(f"{Color.YELLOW}Connected to database{Color.END}")
# return self.connection
# except (Exception, psycopg2.Error) as error:
# logging.error(f"{Color.RED}Error while connecting to PostgreSQL{Color.END}")
# logging.error(error)
# return
def __init__(self, uri, user, pwd):
logging.info(f"{Color.YELLOW}Connecting to database...{Color.END}")
self.__uri = uri
self.__user = user
self.__pwd = pwd
self.__driver = None
try:
self.__driver = GraphDatabase.driver(self.__uri, auth=(self.__user, self.__pwd))
logging.info(f"{Color.YELLOW}Connected to database{Color.END}")
except Exception as error:
logging.error(f"{Color.RED}Error while connecting to Neo4j{Color.END}")
logging.error(error)
def close(self):
if self.__driver is not None:
self.__driver.close()
def query(self, query, parameters=None, db=None):
assert self.__driver is not None, "Driver not initialized!"
session = None
response = None
try:
session = self.__driver.session(database=db) if db is not None else self.__driver.session()
logging.info(query)
response = list(session.run(query))
except Exception as error:
logging.error(error)
logging.error(f"{Color.RED}Query failed{Color.END}")
finally:
if session is not None:
session.close()
return response
class Database:
def __init__(self):
"""Connect to database"""
self.config = json.load(open("test_config.json", "r"))
self.connection = Connection(
uri = self.config.get("gdb").get("uri"),
user = self.config.get("gdb").get("user"),
pwd = self.config.get("gdb").get("password"),
)
def get_knowzee_data(self):
"""Get data from knowzee table"""
block_name = []
block_short_description = []
block_long_description = []
color_comment = []
query = "Match (n:aaa) return n;"
dataset = self.connection.query(query,db="neo4j")
dataset = {
"NAME": "whether a risk is unjustifiable",
"SHORT DESCRIPTION": "Whether a risk is unjustifiable depends on if it is a gross deviation from the standard of care.",
"LONG DESCRIPTION": "In evaluating whether a risk is unjustifiable or substantial depends on whether it is a gross deviation from the standard of care.",
"PICTURES": "",
"PEOPLE": "",
"PLACES": "",
"PROFESSIONS": "",
"THINGS": "",
"TIMES": "",
"RESULTS": "",
"ACTIONS": "",
"GROUPS": "",
"CONNECTIONS": "",
"PRIORITY WITHIN GROUPS": "",
"PRIORITY FOR ALL BLOCKS": "",
"SORT ORDER": "",
"RELATED KEY WORDS": "",
"COLOR COMMENT": "Color comment for block 1"
}
for row in dataset:
block_name.append(row[0])
block_short_description.append(row[1])
block_long_description.append(row[2])
color_comment.append(row[-2])
return (
block_name,
block_short_description,
block_long_description,
color_comment,
)
def get_knowzee_metadata(self, query: str = "SELECT * FROM knowzee"):
"""Get metadata from knowzee table"""
name = []
title = []
description = []
pass
and I use uri as follow. "uri": "neo4j+ssc://localhost:7687",
But when I run this code in docker, i get error message as this:
Unable
to retrieve routing information
when I run query on browser it works. What is issue?
I solved issue. I wrote docker container name instead of "localhost" Thank you.
Glad to hear you found a solution. Before I read that message, I started to skim your code and spotted that you try
/catch
around the driver initialization. Please note that this is a lazy operation.
Driver objects only open connections and pool them as needed. To verify that the driver is able to communicate with the database without executing any query, use
neo4j.Driver.verify_connectivity()
. -- API docs
Hi @robsdedude ,
I'm facing a similar issue on a Neoj4 instance 5.8 (4.4 shows same behavior) deployed on a Kubernetes cluster. A python script inserts data to the database. After a certain amount of time, the insertion process suddenly starts to fail sporadically with
neo4j.exceptions.ServiceUnavailable: Unable to retrieve routing information
Below, a "bad/good-case comparison" of the debug log.
Any ideas/recommendations how to fix/analyse the issue?
Looking forward to receive expert feedback 😊!
Hi @hafeja,
For support I generally refer to either of
I think the online community would be the best place for this.
Now I still want to give some insight of what I see in the logs:
While committing one of the transactions the server replies with Neo.TransientError.Database.DatabaseUnavailable
. You can find that error here where it says
The database is not currently available to serve your request, refer to the database logs for more details. Retrying your request at a later time may succeed.
So check your database logs and see if you can spot something interesting there. I'm not sure what conditions would lead the DBMS to respons with such an error, for that I don't know enough about the server internals.
After that initial error, the driver retires the transaction because the error is classified as TransientError
. That means your using transaction functions or driver.execute_query
:+1: that's good. The second attempt however fails immediately because the driver picks a dead connection from the pool (OSError("No data")
). It's hard to say what killed the connection. If you have for instance an aggressive load balancer in your network that terminates TCP connections that have been idle for a while (most cloud providers like AWS and GCP to that for example), I recommend lowering the max_connection_lifetime
which will make the driver never use a connection that has been open for longer than the configured threshold.
Hi @robsdedude ,
trying different values of max-connection-lifetime
doesn't help in my case.
However, increasing max-transaction-retry-time
to ~3 minutes does solve my issue.
Thanks for your fast and helpful hints 👍
hello@robsdedude I am trying to connect to my Neo4j database, but I encountered an error. Could you assist me?I will be very grateful.
os.environ["OPENAI_API_KEY"] = "***" os.environ["NEO4J_URI"] = "neo4j+ssc://c48f666c.databases.neo4j.io" os.environ["NEO4J_USERNAME"] = "neo4j" os.environ["NEO4J_PASSWORD"] = "***"
Failed to read from defunct connection ResolvedIPv4Address(('34.66.78.163', 7687)) (ResolvedIPv4Address(('34.66.78.163', 7687))) Unable to retrieve routing information
@shuxdh as written above:
For support I generally refer to either of
- Neo4j Online Community: https://community.neo4j.com/
- StackOverflow: https://stackoverflow.com/questions/tagged/neo4j+python
- Discord: https://discord.gg/neo4j
- Neo4j Support Portal if you are an enterprise customer: https://support.neo4j.com/
I think the online community would be the best place for this.
There, you should also include additional information like what you're trying to achieve, code snippets (if applicable), what driver language and version, what server version, stack traces of errors (if applicable), driver debug logs (see the API docs on how to enable them).
This is occurring in a databricks Python notebook when using the neo4j python driver:
from neo4j import GraphDatabase
I receive the following error:Here is my code for connecting to neo4j:
This code successfully writes data to a different neo4j database when tested