ibmdb / node-ibm_db

IBM DB2 and IBM Informix bindings for node
MIT License
188 stars 151 forks source link

conn.query callback being fired before all queries are finished. (possibly) #982

Closed Jarmahent closed 4 months ago

Jarmahent commented 4 months ago

I am building a simple file upload web server for someone from my company to be able to insert over 100k rows into a table in our DB all at once.

Everything works fine except for the below logic. I am taking a csv and turning it into inserts and then i am doing a .forEach for each batch of joined inserts (10k per batch at the moment)

I have logic inside of a forEach to check if the current batch we're on is the last batch, if so close the connection and send back a 200.

The issue is that, though the inserts do occur, if i submit a file of 20k records the inserts only get up to around 11k before the connection is closed.

For some reason the conn.close() command is being executed before all the records have been inserted.

I can verify this because once i remove the conn.close() code, all 20k records get inserted correctly.

Is there something i am doing wrong or is it a bug? I cannot tell.

            chunkedInserts.forEach((chunkedInsert,index) =>{
              let joinedInserts = chunkedInsert.join('');
              conn.query(joinedInserts, function (err, data) {
                if (err){console.error(err);}
                if(index===chunkedInserts.length-1){
                  conn.close();
                  return res.status(200).json({"status": "records inserted"})
                }
              });
            });
Jarmahent commented 4 months ago

To answer my own question i was doing something wrong.

Switching over to using async await fixed the issue:

          conn.query('DELETE FROM MY_TABLE', async function (err, data) {
            if (err){console.error(err);}
            for(var insert of chunkedInserts){
              let joinedInserts = insert.join('');
              await conn.query(joinedInserts);

              if (err){console.error(err);}
              if(insert===chunkedInserts[chunkedInserts.length-1]){
                await conn.close()
                return res.status(200).json({"status": "records inserted"})
              }
            }
        });