Value cannot be null. (Parameter 'source') on ParseInboundEmailWebhookAsync (.NET 5)

[Sn3b]

I keep getting the error Value cannot be null. (Parameter 'source') and I'm not sure where to look. It happens on this line:

var inboundEmail = await parser.ParseInboundEmailWebhookAsync(Request.Body).ConfigureAwait(false);

Please help :)

[Sn3b]

btw I do not have the option POST the raw, full MIME message checked.

[Jericho]

Are you able to capture the content of the payload posted to you by SendGrid? This would be useful in order to attempt to reproduce your situation.

btw I do not have the option POST the raw, full MIME message checked.

Thanks for confirming. That's exactly what I was about to ask you.

[Sn3b]

@Jericho thank you for your quick reply, here is what I was able to extract:

POST https://dev-api.test-domain.com/api/hooks/mail HTTP/1.1
Connection: Keep-Alive
Content-Length: 3989
Content-Type: multipart/form-data; boundary=xYzZY
Host: dev-api.test-domain.com
Max-Forwards: 10
Te: deflate,gzip;q=0.3
User-Agent: Sendlib/1.0
X-WAWS-Unencoded-URL: /api/hooks/mail
CLIENT-IP: xxx.xxx.xxx.xxx:xxxxx
X-ARR-LOG-ID: aed22c5e-a92e-45dc-b7f8-dde7e96beb1c
DISGUISED-HOST: dev-api.test-domain.com
X-SITE-DEPLOYMENT-ID: azure-app-service-name
WAS-DEFAULT-HOSTNAME: azure-app-service-name.azurewebsites.net
X-Original-URL: /api/hooks/mail
X-Forwarded-For: xxx.xxx.xxx.xxx:xxxxx
X-ARR-SSL: 2048|256|C=US, S=Arizona, L=Scottsdale, O="GoDaddy.com, Inc.", OU=http://certs.godaddy.com/repository/, CN=Go Daddy Secure Certificate Authority - G2|OU=Domain Control Validated, CN=*.test-domain.com
X-Forwarded-Proto: https
X-AppService-Proto: https
X-Forwarded-TlsVersion: 1.2

--xYzZY
Content-Disposition: form-data; name="headers"

Received: by mx0154p1mdw1.sendgrid.net with SMTP id 78zHhW5e6h Wed, 16 Feb 2022 08:52:34 +0000 (UTC)
Received: from mail-ed1-f47.google.com (unknown [209.85.208.47]) by mx0154p1mdw1.sendgrid.net (Postfix) with ESMTPS id 3D9ED260CF5 for <company-name@dev.ontest-domain.com>; Wed, 16 Feb 2022 08:52:34 +0000 (UTC)
Received: by mail-ed1-f47.google.com with SMTP id t21so2695040edd.3 for <company-name@dev.ontest-domain.com>; Wed, 16 Feb 2022 00:52:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=test-domain.com; s=google; h=mime-version:from:date:message-id:subject:to; bh=GeESplkO8VSf6Bo2nRUA+Y6XAJ2BzLdA3OIEfnJx7Zw=; b=RK31dUIYKQuUPCy3lvnGG+fsJH8B9Cbluk3Owcp+OT40daS2IK6ynehRB84iSZPrdg eN48ZHNSkBUIrTcnFWhjpM/mo/JzObngp3aRLkeW7gFneOJhPawuyiI6W6GVtfqJiUO7 PabrWFJaqOkz1EUUTVgTLdj9sa8v1j/6iz8Y/QjDjaoDB+KziBOMI7fTja3zBL67Mp33 NyjiGI0P7Ak10pc3I23oHaI6BO2abIjLSL/YLzzmqtZa5mTgr4LHOgdXzlSAuWIdZAhT 9f3AwRxNT8ret1M1GfUTiR/Zh6hl+4VM4tpigSSAKyhBkgeUxJN9m1nM3c5UvGAIMt+l mSLQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=GeESplkO8VSf6Bo2nRUA+Y6XAJ2BzLdA3OIEfnJx7Zw=; b=y/cLJs8atIL8wobxxac0WrOKyv6JhN9SFD76PSy4lMensXHvuRUGSt7EfSmI4gEqn7 HqH1+uc2R36CCVAJNS/bq6pwTBsPnqg7BvuCGdKkd4xU9boetFwTT91HNaw0vSRb77GM VWwMibkvDpUdfYopOTlxgi3ighD7FPoKldOP+Wxh1XXnGws8RZcXECUp5wTz5+wsFnHE ShRqNustRpBcy+9+qj+/p+9GfrlseGtnP8fCzqnUCxF+/G4K54EDnpUpCs6dCwHuTeYZ Jz0EInKhRcATu+cL5G3iqYNOGYcIFuYj+2wsXMdV5tcNldmt39Y2cXiukvGsGgsHOU3k oiWw==
X-Gm-Message-State: AOAM532/3SBXSvqw2Nc5WlkGFlJfhl1scrkEBBQ3edC2AFJO9sWeCbWr uFvo7wJmRKkDU8vVzrg7n5nOHpiBFp2WYK1z9LDJRlhtxNw=
X-Google-Smtp-Source: ABdhPJxHsZsbaFwiT88EaTN1DK3HbfKqCe+cRGzYr25tK96C/eBLTGooZONNfJH4PRaIkhb6uKt7rJY1i3jCOyQt93I=
X-Received: by 2002:a05:6402:430f:b0:410:a082:c6da with SMTP id m15-20020a056402430f00b00410a082c6damr1793486edc.438.1645001553122; Wed, 16 Feb 2022 00:52:33 -0800 (PST)
MIME-Version: 1.0
From: Firstname Lastname <my-addy@test-domain.com>
Date: Wed, 16 Feb 2022 09:52:22 +0100
Message-ID: <CAOfevCD6+rrSzCm3zNevhdVzRNdYUQZceCM=FZRTKWEo9H_UaA@mail.gmail.com>
Subject: test
To: company-name@dev.ontest-domain.com
Content-Type: multipart/alternative; boundary="0000000000003ab0b105d81ec507"

--xYzZY
Content-Disposition: form-data; name="dkim"

{@test-domain.com : pass}
--xYzZY
Content-Disposition: form-data; name="to"

company-name@dev.ontest-domain.com
--xYzZY
Content-Disposition: form-data; name="html"

<div dir="ltr">test</div>

--xYzZY
Content-Disposition: form-data; name="from"

Firstname Lastname <my-addy@test-domain.com>
--xYzZY
Content-Disposition: form-data; name="text"

test

--xYzZY
Content-Disposition: form-data; name="sender_ip"

209.85.208.47
--xYzZY
Content-Disposition: form-data; name="spam_report"

Spam detection software, running on the system "mx0154p1mdw1.sendgrid.net",
has NOT identified this incoming email as spam.  The original
message has been attached to this so you can view it or label
similar future email.  If you have any questions, see
@@CONTACT_ADDRESS@@ for details.

Content preview:  test test [...] 

Content analysis details:   (0.0 points, 5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
 0.0 HTML_MESSAGE           BODY: HTML included in message
 0.0 T_MIME_NO_TEXT         No (properly identified) text body parts

--xYzZY
Content-Disposition: form-data; name="envelope"

{"to":["company-name@dev.ontest-domain.com"],"from":"my-addy@test-domain.com"}
--xYzZY
Content-Disposition: form-data; name="attachments"

0
--xYzZY
Content-Disposition: form-data; name="subject"

test
--xYzZY
Content-Disposition: form-data; name="spam_score"

0.011
--xYzZY
Content-Disposition: form-data; name="charsets"

{"to":"UTF-8","html":"UTF-8","subject":"UTF-8","from":"UTF-8","text":"UTF-8"}
--xYzZY
Content-Disposition: form-data; name="SPF"

pass
--xYzZY--

[Jericho]

I tested the payload you provided and I can parse it, no problem:

I copied the payload you provided (the payload starts at the first "--xYzZY") to a text file and ran the following simplistic unit test. The two assertions pass successfully:

[Fact]
public async Task sn3b()
{
    using (var fileStream = File.OpenRead("c:\temp\sn3b.txt"))
    {
        var parser = new WebhookParser();
        var inboundEmail = await parser.ParseInboundEmailWebhookAsync(fileStream).ConfigureAwait(false);

        Assert.NotNull(inboundEmail);
        Assert.Equal("my-addy@test-domain.com", inboundEmail.From.Email);
    }
}

So, unless you can provide clear steps to reproduce the problem you are facing, I'm afraid I won't be able to help.

I know you already confirmed that POST the raw, full MIME message is unchecked in your environment (and the payload you provided seems to confirm it), but I still encourage you to double check this setting. In my experience, it's the root cause of 99.999999999999% of all problems related to parsing inbound emails. Also, a lot of people copy and paste the sample payload that SendGrid provides in their documentation (for testing purposes) but what they don't realize is that this sample is in the "raw" format, therefore StrongGrid is not able to parse it.

[Sn3b]

Hi @Jericho,

Thanks for your investigation, I am currently trying to figure it out, and it seems the issue is in Http-Multipart-Data-Parser. It happens on this line where line is null.

Btw, the extract I sent you is the actual output from a real SendGrid request that I logged and formatted using this extention, so indeed as the author says it might not be 100% correct.

[Jericho]

Http-Multipart-Data-Parser is the component that parses the payload. It will throw an exception if your payload is not a proper HTTP multipart response such as, for example, if it's in SendGrid's "raw" format.

I also vaguely remember another user having difficulty parsing their payloads which were posted on a Unix environment using a command line tool (probably 'curl' if I remember correctly). I'm going from memory so I could be wrong on some of the details but I'm pretty sure the issue turned out to be that this command line tool was using a different line separator which Http-Multipart-Data-Parser did not support. Again, the details are very vague in my memory so take this with a grain of salt.

[Sn3b]

That's actually very good info. I'm trying to parse from an azure app service, and I know that when using the log stream to see (almost real time) logs, carriage returns are missing... I'll continue my investigations when the kids are in bed, I'll let you know if I find anything. Thanks a lot for your time, it's greatly appreciated!

[Sn3b]

Hi @Jericho, just to let you know that my problem has disappeared. I feel a bit stupid because I don't know what started it in the first place, but I know I made it worse by investigating. I didn't realise the snippet I used to output the raw request didn't reset the body so the stream could be read again, which explains why it was empty. Your intervention however was of great help, please do continue to support your project with as much fervour.

Thanks!

[Jericho]

I'm really glad you figure it out.

Thanks for the kind words!

[Sn3b]

AHA! I think I might have found what caused my issue in the first place, I had a cancellation token in my signature as follows:

public async Task< IActionResult > Mail( CancellationToken cancellationToken = default ) { .. }

I just re-added it and the Value cannot be null. (Parameter 'source') error is back.