MimeKit's MimeParser Counts Some MBOX Emails Multiple Times and Splits Messages

[DeepakBisht94]

Describe the bug

I'm using MimeKit's MimeParser in C# to count the number of emails in an MBOX file. While the parser works well for most emails, I've encountered issues where certain emails are counted multiple times or split into two or three parts. Additionally, some emails trigger a "Failed to parse message headers" error, although the primary concern is the inaccurate counting and message splitting.

Here's the relevant portion of my code:

Platform (please complete the following information):

• OS: [Windows]
• .NET Framework: [ .NET 4.7]
• MimeKit Version: 4.7.1

To Reproduce Steps to reproduce the behavior: Download the attached MBOX file and run the following code.

Expected behavior I'm sharing a sample email from the problematic MBOX file below, which seems to cause the parser to count it as two messages. It should be one. Also some emails giving Failed to parse message headers.

Code Snippets

private void ContinueAfterError(Stream stream, MimeParser parser)
{
    long newPosition = parser.Position + 1024; 
    if (newPosition < stream.Length)
    {
        stream.Position = newPosition;
        parser.SetStream(stream, MimeFormat.Mbox);
    }
    else
    {
        Log("Reached end of stream or unable to skip ahead safely.");
    }
}

try
{
    int count = 0;

    if (!File.Exists(LoadFile))
    {
        Log($"File not found: {LoadFile}");
        MessageBox.Show($"The file '{LoadFile}' does not exist.", "File Not Found", MessageBoxButtons.OK, MessageBoxIcon.Error);
        return;
    }

    using (var fileStream = File.OpenRead(LoadFile))
    {
        var parserOptions = new ParserOptions(); 
        var mboxParser = new MimeParser(parserOptions, fileStream, MimeFormat.Mbox);

        while (!mboxParser.IsEndOfStream)
        {
            try
            {
                var message = mboxParser.ParseMessage();
                if (message != null)
                {
                    count++;
                    Log($"Email {count}: Subject - {message.Subject}");
                    Log($"MBOX marker at message {count}: {mboxParser.MboxMarker}");
                }
            }
            catch (Exception parseEx)
            {
                Log($"Failed to parse message: {parseEx.Message}");
                Log($"Error occurred near MBOX marker: {mboxParser.MboxMarker}");
                ContinueAfterError(fileStream, mboxParser);
            }
        }
    }

    _totalCount = count;
    UpdateLabelCount(count);
}
catch (Exception ex)
{
    MessageBox.Show($"An error occurred while reading the file: {ex.Message}", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
}

Issue Files.zip

[jstedfast]

The problem is an improperly formatted mbox file. There's no way to parse this the way you would expect because there are From lines in the middle of the message bodies. This isn't solvable by any mbox parser.

https://stackoverflow.com/questions/79045977/mimekits-mimeparser-counts-some-mbox-emails-multiple-times-and-splits-messages

There are a few comments that others posted which are correct - the From lines in the message body need to be escaped or encoded:

>From topics you know about

-or-

=46rom topics you know about

If you expected the entire mbox from Thunderbird (as opposed to creating the mbox file yourself via concatenation of multiple messages with your own From lines), then you should file a bug report against Thunderbird.