Getting error "expecting POST /_api/cursor" from Arangojs

[DougGarno55]

If I try to get records greater than 1,000, I get the "expecting POST /_api/cursor" error when I call the cursor.all() method in ArangoDB.js.

Here is my installation: ArangoDB version 3.11.3 ArangoDB.js versioin 8.6.0 Node.js version 18.14.0

If I add 'LIMIT 1000' to the query, everything works as expected. If I add 'LIMIT 1001' or anything over 1,000 I get the error. My Test database has one collection (FOICU) with 5,000 documents and only one attribute (CU_NAME).

To replicate:

const Database = require( 'arangojs' ).Database;
let db = new Database({
   url: 'HTTP://127.0.0.1:8529',
   databaseName: 'TEST',
   auth: { username: 'root', password: 'test' }
});
let cursor = await db.query( 'FOR doc1 IN FOICU RETURN { "CU_NAME": doc1.CU_NAME }' );
let result = await cursor.all();
return result;

I've tried this on multiple databases, multiple collections all with the same result.

[DougGarno55]

Release 8.3.0 works, the problem started with release 8.4.0.

[pluma4345]

ArangoDB 3.7 reached its End of Life in May 2022: https://arangodb.com/subscriptions/end-of-life-notice/

Please upgrade to a more recent (and supported) version of ArangoDB or continue using arangojs 8.3.0 until you are able to upgrade.

If you can replicate this problem with ArangoDB 3.10 or 3.11, please feel free to re-open this issue.

[DougGarno55]

I tested with ArangoDB 3.11.6 and the JavaScript driver still reports the same problem. This has not been resolved! I can't seem to re-open this issue.

[pluma4345]

For the life of me I can't replicate this with ArangoDB 3.11 and the latest driver version. Note that the driver changed from PUT to POST in a recent release so the message does not make any sense. Are you sure that the code you are running is using the driver version you think you are using? if it's using an older version (e.g. via a nested dependency) that would explain the error.

[pluma4345]

FWIW I'm using the following to replicate this:

const c = await db.query('FOR i IN 1...10000 RETURN i');
const r = await c.all();
console.log(r);

This results in multiple requests to fetch consecutive batches, so it would trigger this error.

[DougGarno55]

I used NPM to install the driver in a NodeJS server. I checked the package.json file in the node_modules/arangojs/ folder for the version. Any version after 8.3.0 will have this error. I have not tracked down the actual line that throws the error, but I guess that I need to. I have several servers using the arangojs, and they are all using pre 8.4.0 which is the version that the error first occurred. I will debug the query and step into the driver source to find the error.

[DougGarno55]

I've tracked it down to line 435 of the connection.js file. Here it is building the Promise for the request. The default method is "GET" and I couldn't tell if a method was specified or not and the Promise is run and throws the error during execution. It is on the second iteration of the flatmap method in the cusror.js file, when it is callin the _more method at line 78. Line 81 of cursor.js does use the POST method on the request, but I think somewhere along the line the method is lost and the default in the connection.js file uses GET. There are a lot of places where the default uses GET within the entire code base and if POST is required, then there could be issues where the method gets lost and the default uses GET. I ran out of time this morning, I'll try to track down more information if needed.

[DougGarno55]

I added precaptureStackTraces when creating the Database and this is the error information that I got:

"ArangoError: expecting POST /_api/cursor
    at new ArangoError (/Users/douglasgarno/Desktop/Dev/ClearCore/CLite/node_modules/arangojs/error.js:114:21)\n
    at callback (/Users/douglasgarno/Desktop/Dev/ClearCore/CLite/node_modules/arangojs/connection.js:211:31)\n
    at IncomingMessage.<anonymous> (/Users/douglasgarno/Desktop/Dev/ClearCore/CLite/node_modules/arangojs/lib/request.node.js:116:21)\n
    at IncomingMessage.emit (node:events:525:35)\n
    at endReadableNT (node:internal/streams/readable:1359:12)\n
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)\n
    at Connection.request (/Users/douglasgarno/Desktop/Dev/ClearCore/CLite/node_modules/arangojs/connection.js:436:16)\n
    at Database.request (/Users/douglasgarno/Desktop/Dev/ClearCore/CLite/node_modules/arangojs/database.js:245:33)\n
    at BatchedArrayCursor._more (/Users/douglasgarno/Desktop/Dev/ClearCore/CLite/node_modules/arangojs/cursor.js:81:37)\n
    at BatchedArrayCursor.next (/Users/douglasgarno/Desktop/Dev/ClearCore/CLite/node_modules/arangojs/cursor.js:228:24)\n
    at BatchedArrayCursor.flatMap (/Users/douglasgarno/Desktop/Dev/ClearCore/CLite/node_modules/arangojs/cursor.js:387:45)\n
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"

Has something to do with batches and the hasNext flag (I think). When in the _more() method of cursor.js line 78 is where the exception is thrown. Not sure what more I can do on my end. But if I roll back the arangojs driver to 8.3.0 everything works as expected. Let me know if there is something else I can do to give you more information.

[pluma4345]

Can you modify your local copy of the module to log the request method at each step? If the request that is sent out uses the wrong method, it should change at some point.

[DougGarno55]

Where the error occurs, connection.js line 436, the method is always POST. The first time everything works as expected and the Promise is resolved with the first 1,000 records. The second time is when the error happens. The Promise builds a task and here is the task values:

I wonder if the error message is throwing us off and it is telling us the correct method. The error has something to do with the batch cursor getting the next group of records.

[DougGarno55]

The second time the request is returned from the server, connection.js line 163, the server returns a status of 400, statusMessage = "Bad Request". Here is the request (I have to take snapshots, sorry):

[pluma4345]

The screenshots seem to confirm arangojs sends a POST request. It also looks like the ArangoDB deployment is running locally.

My best guess would be that something between arangojs and ArangoDB messes with the request but I'm not sure what that would be.

[DougGarno55]

This only started with version 8.4.0 of the driver when you went from PUT to POST. If I use version 8.3.0 of the driver everything works as expected, so I don't think there is anything external affecting the request.

[sysdynamix]

@DougGarno55 Thanks for your diagnostics and workaround. I have experienced the exact issue and downgrading the arangojs driver to 8.3.0 fixes the problem for now. Hopefully there is a fundamental fix soon.