ArangoDB-Community / pyArango

Python Driver for ArangoDB with built-in validation
https://pyarango.readthedocs.io/en/latest/
Apache License 2.0
238 stars 90 forks source link

pyArango sockets crashing on high write load on Windows 10 #189

Closed saliksyed closed 4 years ago

saliksyed commented 4 years ago

I'm trying to stress test the database for an application I'm working on, which uses 3.6.2 community edition , to do this I need to create an initial dataset. Unfortunately Arango falls over during the creation ( I create ~10k objects, for each object I set about 20-30 properties, calling doc.save() after updating each property). I am using a singleton connection class that wraps the Connection object (I also tried making a pool of multiple connections but that had no effect)

requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8529): Max retries exceeded with url: /_db/imvela/_api/document/measurement/3001960?collection=measurement&waitForSync=False (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000002BA7E57E388>: Failed to establish a new connection: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted'))

When I try to access the arango web interface it has also crashed (it usually comes back up after a few seconds -- indicating to me that perhaps the server restarted?). I checked the logs and there is no indication of any error.

I feel like the rate of writes is not so high to warrant the database shutting down :(

I even added a time.sleep call and checked the writes/sec and it's < 500 writes / sec image

saliksyed commented 4 years ago

Note : the error does not occur when running Arango on a google cloud VM (Debian 9) n1-standard-2 (2 vCPUs, 7.5 GB memory)

saliksyed commented 4 years ago

I gave up :/ -- now the DB doesn't even respond on my machine. Works great on Mac OSX,

romoxi commented 3 years ago

Same problem with 3.7.10 community edition on my Win 10.

Is there a solution?

romoxi commented 3 years ago

I use Python-Arango instead without any problems