Inconsistent `mSet`: Connection Reset

[maplenk]

Steps to reproduce

Hi, I am trying to dump some data on dicedb instance hosted on GCP via NodeJS + Redis

Node Version: 20 DiceDB Version: Latest (checked out from main branch 2 days ago) Node Redis Package: https://www.npmjs.com/package/redis - 4.7.0

Steps:

1. Get data from DB (Attached the data) table1.csv table2.csv
2. Iterate over it and dump to redis in batches via mSet

Code:

        // Initialize objects to store tickets and sales orders
        const tickets = {};
        const salesOrders = {};

        // Iterate over each row in the result set
        for (const row of ticketrows) {
            // Create a ticket key using the uniqueReferenceID
            const ticketKey = `${row.uniqueReferenceID}`;

            // Trim the ticket data to only include the necessary fields
            const currentTicket = {
                tkt: ticketKey,
                bvID: row.batchVariantID,
                bvName: row.batchVariantName,
                value: row.productValue,
            };
            const ticketData = {
                so: row.salesOrderID,
                qty: 1,
                tkts: [currentTicket],
            };

            // Store the ticket data as a JSON string in the tickets object
            tickets[ticketKey] = JSON.stringify(ticketData);
        }

        // Iterate over each row in the result set
        for (const row of sorows) {
            // Create a ticket key using the uniqueReferenceID
            const ticketKey = `${row.uniqueReferenceID}`;

            // Trim the ticket data to only include the necessary fields
            const currentTicket = {
                tkt: ticketKey,
                bvID: row.batchVariantID,
                bvName: row.batchVariantName,
                value: row.productValue,
            };
            const ticketData = {
                so: row.salesOrderID,
                qty: 1,
                tkts: [currentTicket],
            };

            // Create a sales order key using the salesOrderID and batchVariantID
            const salesOrderKey = `${row.salesOrderID}`;

            // Check if the sales order already exists in the salesOrders object
            if (salesOrders[salesOrderKey] !== undefined) {
                // Increment the quantity and add the ticket to the sales order
                let tempSalesOrder = JSON.parse(salesOrders[salesOrderKey]);
                tempSalesOrder.qty++;
                tempSalesOrder.tkts.push(currentTicket);
                salesOrders[salesOrderKey] = JSON.stringify(tempSalesOrder);
            } else {
                salesOrders[salesOrderKey] = JSON.stringify(ticketData);
            }
        }

        // Get the count of tickets and sales orders
        let ticketCount = Object.keys(tickets).length;
        let salesOrderCount = Object.keys(salesOrders).length;

        console.log('Active tickets:', ticketCount);
        console.log('Active sales orders:', salesOrderCount);

        // Dump tickets to Redis in batches
        if (ticketCount > 0) {
            const batchSize = 10;
            for (let i = 0; i < ticketCount; i += batchSize) {
                const batch = Object.entries(tickets).slice(i, i + batchSize);
                await redis.mSet(batch);
            }
            console.log('Dumped active tickets to Redis');
            console.log('Dumped all active tickets to Redis in bulk');
        } else {
            console.log('No active tickets to dump');
        }

        // Dump sales orders to Redis in batches
        if (salesOrderCount > 0) {
            const batchSize = 5;
            for (let i = 0; i < salesOrderCount; i += batchSize) {
                const batch = Object.entries(salesOrders).slice(i, i + batchSize);
                await redis.mSet(batch);
            }
            console.log('Dumped active sales orders to Redis');
        } else {
            console.log('No active sales orders to dump');
        }

        // Return true to indicate successful completion
        return true;

3 out of 5 times it works. Other times I get a connection error

DiceDB logs from server:

1:38PM INF FLUSHDB called args={}
1:38PM INF FLUSHDB called args={}
1:39PM INF FLUSHDB called args={}
1:39PM WRN connection timeout

Expected output

The expected output when the above set of commands (maybe when run on Redis)

Works all the time.

Observed output

The observed output when the above set of commands when run on DiceDB

WRN connection timeout

Expectations for resolution

This issue will be considered resolved when the following things are done

1. changes in the dice code to meet the expected behavior
2. addition of relevant test case to ensure we catch the regression

You can find the tests under the integration_tests directory of the dice repository and the steps to run are in the README file. Refer to the following links to set up DiceDB and Redis 7.2.5 locally

• setup DiceDB locally
• setup Redis 7.2.5 locally
• setup DiceDB CLI

[JyotinderSingh]

Thanks for reporting this, we're looking into it. Will keep this thread posted.

[BadriVishalPadhy]

@JyotinderSingh can u plz assign this issue to me

[JyotinderSingh]

@JyotinderSingh can u plz assign this issue to me

Feel free to look into this, but fyi, a few other team members are also investigating this. Please report your findings here.

[KaviiSuri]

@maplenk , quick question, could you provide any info regarding the infra for this setup? do you have both the server and dicedb running on the same machine or different ones?

[maplenk]

Hey, Installed dicedb on Ubuntu 22 on GCP Connecting to it remotely via mac