Bulk insert is losing data

[glebteterin]

Describe the bug Sometimes Bulk Insert returns the result where RowsAffected() is less than we sent (or wanted to insert). For instance, we sent 9506 rows, but RowsAffected() returned 5839.

It doesn't happen often, maybe once every six months. But when it happens, if we run the same code again, the expected number of rows will be inserted successfully, hence the data that we insert is valid. There are no errors returned also, so no way we can reproduce it.

I'm open to any ideas/suggestions of how to get to the bottom of this.

Below is example of code:

funk bulkImport(tableName string, dt *datatable) (int64, error) {

    log.Println("Inserting rows", len(dt.rows)) // prints Inserting rows 9506

    columns := dt.PrepareColumns()

    _, err := repo.db.Exec("select 1")
    if err != nil {
        return 0, errors.Wrap(err, "error checking connection")
    }

    stmt, err := repo.db.Prepare(mssql.CopyIn(tableName, mssql.BulkOptions{}, columns...))
    if err != nil {
        return 0, errors.Wrap(err, "error preparing bulk import statement")
    }

    defer func() {
        if err := stmt.Close(); err != nil {
            repo.logger.Error("error closing bulk import statement", err)
        }
    }()

    for i := range dt.rows {
        _, err = stmt.Exec(dt.GetValuesLimited(i)...)
        if err != nil {
            dbStats := repo.db.Stats()
            return 0, errors.Wrapf(err, "error adding record. open: %v", dbStats.OpenConnections)
        }
    }

    result, err := stmt.Exec()
    if err != nil {

        dbStats := repo.db.Stats()
        return 0, errors.Wrapf(err, "error executing bulk import. open: %v", dbStats.OpenConnections)
    }

    insertedRowCount, _ := result.RowsAffected()

    return insertedRowCount, nil // insertedRowCount = 5839
}

Further technical details

SQL Server version: Microsoft SQL Azure (RTM) - 12.0.2000.8 Operating system: Docker container