agmen-hu / node-datapumps

Node.js ETL (Extract, Transform, Load) toolkit for easy data import, export or transfer between systems.
MIT License
291 stars 38 forks source link

Error while insert item in MySql #39

Closed rafaneri closed 7 years ago

rafaneri commented 7 years ago

Hello,

I have a problem when I try to get data from a MongodbMixin to insert in a MysqlMixin. I'm getting the error:

Error in pump (root): Error: ER_DUP_ENTRY: Duplicate entry 'EMAIL@HOTMAIL.COM' for key 'UQ_Contat_contact_email' Unhandled rejection PumpingFailedError: Pumping failed. See .errorBuffer() contents for error messages

I don't understanding the error message "Unhandled rejection PumpingFailedError: Pumping failed. See .errorBuffer() contents for error messages".

Can you help me?

Follow my code:


var
    mysql = require('mysql'),
    datapumps = require('datapumps'),
    Pump = datapumps.Pump,
    MongodbMixin = datapumps.mixin.MongodbMixin,
    MysqlMixin = datapumps.mixin.MysqlMixin,
    pump = new Pump();

var connection = mysql.createConnection({
    host: 'localhost',
    user: 'database',
    password: 'user',
    database: 'password'
})

pump
    .mixin(MongodbMixin('mongodb://localhost/database'))
    .useCollection('client')
    .from(pump.find({}))

    .mixin(MysqlMixin(connection))

    .process(function (client) {
        return this.query(`INSERT INTO contact \
            (contact_name,contact_email) VALUES (?)`,
            [client.name, client.contact.email]);
    })
    .logErrorsToConsole()
    .run()
    .then(function () {
        console.log("Done writing contacts to file");
    });
novaki commented 7 years ago

Hello @rafaneri

The error message just wants to say that the pump failed, and the error messages are available in pump.errorBuffer().getContents(). However, as .logErrorsToConsole() was called, error messages are also printed to the stdout. So in fact the only error is the mysql duplicate key error, which came from the mysql driver (seems like you have unique constraint in mysql but not in mongo).