Problem fetching many mails with their attachments

[henrikengelbrink]

Hello, I'm having a problem fetching many mails with their attachments. It seems my script is freezing while loading the mails. Is it possible that nodeJS is to slow for loading the attachments for hundreds of mails ?

This is the code I'm using to fetch the mails and their attachments:

imaps.connect(config).then(function (connection) {

    connection.openBox('INBOX').then(function () {

        var days = 30;
        var hours = 24 * days;
        var delay = hours * 3600 * 1000;
        var currentDate = new Date();
        currentDate.setTime(Date.now() - delay);
        currentDate = currentDate.toISOString();
        var searchCriteria = ['ALL', ['SINCE', currentDate]];
        var fetchOptions = { bodies: ['HEADER.FIELDS (FROM TO SUBJECT DATE)'], struct: true };

        return connection.search(searchCriteria, fetchOptions);
    }).then(function (messages) {
        console.log("### Count: " + messages.length);
        var mails = [];
        var lastMailIDWithAttachments = idOfLastMailWithAttachments(messages);
        messages.forEach(function (message) {
            var subject = "### NIL ###";
            var date = "### NIL ###";
            var sender = "### NIL ###";
            var uid = message.attributes.id;

            message.parts.forEach(function(part)
            {
                subject = part.body.subject[0];
                date = part.body.date[0];
                sender = part.body.from[0];
            })

            var parts = imaps.getParts(message.attributes.struct);
            console.log(uid + " - " + parts.length);

            var attachments = [];
            parts.forEach(function(part)
            {
                if (part.disposition)
                {
                    if (part.disposition.type.toLowerCase() == "attachment")
                    {
                        if (part.disposition.params.filename)
                        {
                            console.log(part.disposition.params.filename);
                            connection.getPartData(message, part).then(function (partData) {
                                // var fullFileName = sender + " ###_### " + part.disposition.params.filename;
                                // fullFileName = fullFileName.split("\"").join("");
                                // fullFileName = fullFileName.split("\'").join("");
                                // fullFileName = fullFileName.split("\/").join("");
                                // fullFileName = fullFileName.split(" ").join("_");
                                var attachment = {
                                    id:part.partID,
                                    filename:part.disposition.params.filename,
                                    sender:sender
                                };

                                var dir = '../PDF_Bills';
                                if (!fs.existsSync(dir))
                                {
                                    fs.mkdirSync(dir);
                                }
                                fs.writeFile(dir + "/" + attachment.filename, partData, function (err) {
                                  if (err) return console.log(err);
                                });
                                attachments.push(attachment);

                                var partIndex = parts.indexOf(part);
                                if (partIndex === parts.length-1)
                                {
                                    var mailObject = {
                                        subject:subject,
                                        date:date,
                                        did:und,
                                        attachments:attachments
                                    };
                                    mails.push(mailObject);
                                    if (lastMailIDWithAttachments === mailObject.uid)
                                    {
                                        connection.end();
                                        console.log("FINISHED");
                                        var jsonString = JSON.stringify(mails);
                                        fs.writeFile("./mails.json", jsonString, function (err) {
                                          if (err) return console.log(err);
                                        });
                                    }
                                }
                            });
                        }
                    }
                }
            });
        });
    });

    connection.once('error', function (err) {
        console.log(err);
    });
});

I hope somebody can help me. Henrik

[chadxz]

It seems my script is freezing while loading the mails

As in you never see the output of this log? console.log("### Count: " + messages.length)

Where in your script do you stop seeing console log output?

[henrikengelbrink]

This is the output in the console:

### Count: 635
154 Mails with Attachment - Last:45728
44558 - 2
44559 - 2
44560 - 7
Bestellformular.pdf
44561 - 2
44562 - 2
44563 - 8
44564 - 2
44565 - 2
44566 - 2
44567 - 19
44568 - 1
44569 - 1
44570 - 3
44571 - 2

If I comment out the parts.forEach-Method and everything in it, the script runs normally.

[chadxz]

you might wanna try running your code under a debugger if you're having trouble figuring out where things are going into the weeds. Webstorm has one built in, or you can just attach something like Node Inspector.

That, or just add more logging! heh

[henrikengelbrink]

I've tested it with node debugger, but the debugger throws no exception, the program stops without any errors.

[chadxz]

Try setting some breakpoints in your code where the program starts to behave incorrectly. Inspect the variables to see if anything looks differently than you would expect in situations where it is supposed to save the file.

[chadxz]

LMK if there's anything else I can do to help troubleshoot.

[melitus]

I have a problem with the email file attachment. When I open the downloaded attachment file will be empty as shown in the screenshot below