Insert large data

[hosseinGanjyar]

I want to Insert large amount of data (~ 8,000,000 records) ,that I get from Sql Server. I use of streaming "mssql" driver for get data.

My problem: When I want to insert data (8,000,000 records), increased memory usage very very much ( > 2GB). Is there a way for insert very very large data using Orientjs???

var ODatabase = require('orientjs').ODatabase;
var db = new ODatabase({
    host: 'localhost',
    port: 2424,
    username: 'root',
    password: '123',
    name: 'test'
});

function get_All_Message(callback) {
    var connection = new sql.Connection(config, function (err) {
        if (err)
            console.log(error.message);

        var request = new sql.Request(connection);
        request.stream = true;

        request.query('select top(8000000) * from Message');
        // ... error checks
        request.on('recordset', function (recordset) {
            // Emitted once for each recordset in a query
        });

        request.on('row', function (message) {
            // Emitted for each row in a recordset
            db.insert().into('Message')
                .set(message)
                .one()
                .then(function (Message) {
                      console.log('Record inserted');
                    }, function (error) {
                        console.log(error.message);
                    }
                )
        });

        request.on('error', function (err) {
            console.log(error.message);
        });

        request.on('done', function (returnValue) {
            // Always emitted as the last one
        });
    });
}

[lvca]

Using OrientJS is not the fastest way to insert data. You could try with ETL by connecting it to MySQL executing your query. It will be much faster.

[StarpTech]

It's because orientjs don't support streaming. Socket connections are really fast. This is an restriction which should be on the roadmap. There are lots of issues according to this.

[hosseinGanjyar]

1- I want Insert data to "OrientDB" database. 2- I want connect to SqlServer, No MySql.

[hosseinGanjyar]

OrientJS wasn't efficient for insert massive data from SqlServer to OrientDB. I used ETL module for massive insert, that is good idea for transpot massive data without increase memory more than 2GB.

Thanks from @saeedtabrizi for help me.