The driver takes lot of time to return data if more than 1000 records

[rajeshvaya]

OrientDB version

2.1.9 community

OrientJS (driver) version

2.1.11

Issue

When a query is run which returns around 800 to 1000 records in the result set the callback function for db.query takes a lot longer

Example a query takes 0.159 sec(s) to run on a console on a remote connection where as fetching the data from the OrientJS driver takes 2.555 sec(s)

Re-produce steps

Dumps

• JSON export
• Database folder

Code

var sql = "SELECT *, in() as _in_CallHistory FROM ( SELECT expand(c) FROM ( SELECT distinct(c) as c FROM ( SELECT out('hasCallHistory') as c FROM agents unwind c ) ) ) limit -1 fetchplan in_*:-2 out_*:-2";

console.time("rawSQL");
db.query(sql, {})
.then(function(records){
    console.timeEnd("rawSQL");
});

Console output

Driver output

[wolf4ood]

hi @rajeshvaya

i've executed it. This is my timing.

rawSQL: 417.963ms

With a single test

Where is OrientDB installed. On the same machine of your app or another?

Is this the timing under some load test?

[rajeshvaya]

Hi @maggiolo00,

OrientDB is installed on a different machine on a local network than the app. The timings which I am getting is on a machine with just OrientDB running.

Should I be changing any configurations or JVM configs on the orientDB's side? I am currently using the default configurations which comes with bin/server.sh

[rajeshvaya]

@maggiolo00, also I just want to make sure that the query fetched 1000+ records for you as well. If you can check with records.length

var sql = "SELECT *, in() as _in_CallHistory FROM ( SELECT expand(c) FROM ( SELECT distinct(c) as c FROM ( SELECT out('hasCallHistory') as c FROM agents unwind c ) ) ) limit -1 fetchplan in_*:-2 out_*:-2";

console.time("rawSQL");
db.query(sql, {})
.then(function(records){
    console.timeEnd("rawSQL");
    console.log(records.length);
});

[StarpTech]

@rajeshvaya

I think the reason is the unoptimized binary driver orientjs.

What the OrientDB Team will do in the next releases?

OrientDB Team will support this project (fix the issues, merge pull requests, etc) starting from today.

The first contribution will be on speeding up the marshalling and unmarshalling activities. By several tests, we found such big bottleneck on performance when Oriento (now orientjs) unmarshalled the result from OrientDB server.

Execution of 100k queries on OrientDB took about 60 seconds, but only 18 seconds (30% of the time) was spent by OrientDB Server + TCP/IP transport. All the rest was taken by Oriento (now orientjs) driver to unmarshall the query result, that is merely 70% of the total time.

• See more at: http://orientdb.com/welcome-to-orientjs/#sthash.xNlexa8F.ZfZwhFbX.dpuf

[StarpTech]

@maggiolo00 @lvca @luigidellaquila What is the status of this goal?

[wolf4ood]

hi @StarpTech i don't know in this case. I think it is not the protocol. The records are lightweight.

@rajeshvaya

the records length is

1447
rawSQL: 332.429ms 

[wolf4ood]

@StarpTech

the native driver is ready

https://github.com/orientechnologies/orientjs-native

This will land with 2.2.x.

I think we will release with the second beta of orientdb 2.2.0 the driver orientjs that uses the binary library.

[StarpTech]

@maggiolo00 great but ready is for me 100% backwards compatible, with integration tests and not only unit tests. When can we expect to use it in 2.1.X or how is the timetable in general?

[rajeshvaya]

@StarpTech, thanks for the info

@maggiolo00, If I need to test the orientjs-native, How do I set it up? Or if you could update the README file it would be great.

I am currently working a project with which I could help out the beta testing of the native unmarshalling with large data set.

[wolf4ood]

@rajeshvaya

it is not released yet. It will the next week i think.

[wolf4ood]

hi @StarpTech

the orientjs native will be used only with 2.2.x.

[rajeshvaya]

@maggiolo00 any update on the 2.2.x release date?