sendgmail: use the GMail API instead of SMTP

[DemiMarie]

This allows using a far more limited OAuth token. sendgmail does not need the ability to delete email, for example.

This doesn’t matter on most systems, where sendgmail runs with the same privilege as the user’s browser. However, it does matter for QubesOS users who want to use sendgmail from a different VM.

[junyer]

Once upon a time, sendgmail did use the Gmail API. However, I had to make it use SMTP instead because that was the only way to stop Gmail munging the plain text. Which is to say, if you want to mail patches, then you don't want to use the Gmail API. T_T

[DemiMarie]

Once upon a time, sendgmail did use the Gmail API. However, I had to make it use SMTP instead because that was the only way to stop Gmail munging the plain text. Which is to say, if you want to mail patches, then you don't want to use the Gmail API. T_T

Did you try the multipart upload API?

[junyer]

Did you try the multipart upload API?

That's not an option for Linux kernel patches, which are sendgmail's raison d'être. See https://static.lwn.net/kerneldoc/process/submitting-patches.html#no-mime-no-links-no-compression-no-attachments-just-plain-text for more information.

[DemiMarie]

The way to go here is to fix Gmail API. LOL, it is 2022!

I mean how the heck can one report a bug in it to Google?

[favonia]

@junyer Sorry that I missed this conversation before making the PR #66. On the other hand, I have managed to use the patched sendgmail (together with git) to send patches to sourcehut and everything looks fine. (So, the arrival of sourcehut means Linux kernel is not the only usage.) Could you possibly point out how Gmail is messing up the plaintext emails? So far I have not seen it, but maybe I was lucky...

[junyer]

The original (internal) report from early April 2019 said:

the patches are not correctly formatted; lines are wrapped and line terminators are being changed (\n -> \r\n).

The reporter attached full details of what they tried to send versus what actually ended up being sent. An illustrative snippet:

@@ -4620,7 +4624,6 @@ static void rt5677_gpio_config(struct rt5677_priv *rt5677, unsigned offset,

@@ -4620,7 +4624,6 @@ static void rt5677_gpio_config(struct
rt5677_priv *rt5677, unsigned offset,

Judging from both the discussion at the time and my notes, switching to SMTP was the only viable solution. If something has changed since then, I would be very keen to know what exactly, but I'm pessimistic that anything has changed in that regard.

[favonia]

@junyer Thanks. I tried sending fake emails with the same headers from git and many long lines (way beyond 78 characters) and lots of spaces. Still didn't observe the hard line breaking or the replacement of line terminators. Could you (or someone in the internal report) possibly try it?

[junyer]

I asked Workspace DevRel to confirm that behaviour change and, if so, whether sendgmail can rely on said behaviour change for the foreseeable future.

[favonia]

@junyer I just rebased my previous code for your reference in case you get a positive response: https://github.com/favonia/fork4pr-gmail-oauth2-tools/tree/reduce-scope

[junyer]

Thanks. Unsurprisingly, it does resemble the code from ~4.5 years ago in terms of invoking the Gmail API. However, note that git send-email actually executes sendmail -i, not sendmail -i -t, so it doesn't specify Bcc: and instead specifies the recipients (To:, Cc:, Bcc:) on the command line. Thus, sendgmail needed to synthesise Bcc: explicitly for the Gmail API.

[favonia]

@junyer Done the Bcc: by a simple conversion of flag.Args(). An address will be added to Bcc: even if it's already in To: because I prefer not to parse the actual mail. :sweat_smile:

PS: actually let me implement the folding in case there are too many recipients... Done!

[junyer]

mail.ReadMessage() makes it straightforward to synthesise Bcc: precisely, so I would want to resurrect that logic, but let's not cross that bridge until we come to it.

[favonia]

@junyer Alright, then we will parse the mail when we come to it. I was trying minimize work. 😅 Parsing the mail will indeed approximate the bcc list better in many cases, but I don't see how we can reconstruct the list precisely when git is not helping us. More precisely, these two lines seem to generate the same arguments to sendmail:

git send-email --to=me@gmail.com
git send-email --to=me@gmail.com --bcc=me@gmail.com

I suppose you are referring to some "minimal" bcc list. I would argue that Bcc: should include me@gmail.com in the second case but maybe it's better to exclude it if we cannot distinguish these two cases.

[junyer]

sendgmail would have exhibited the same behaviour, I suppose, because it used the stringset package to discard the To: and Cc: addresses from flag.Args() and then set Bcc: to the remaining addresses (if any). I agree that there's not much else that's feasible given the SMTP-like semantics (i.e. RCPT TO versus DATA) in play. So many aspects here are Not Ideal™.

[favonia]

@junyer I might have (finally!) encountered a case where Gmail inserted a newline. Strangely, the email looks perfect okay in Gmail itself and it looks correct if I download the original email from the Gmail web interface. However, the receiving end seems to have gotten an email with an extra newline. So, maybe it's just that I have been very lucky so far. Sorry for the noise. 😟

[junyer]

No worries. I'm hoping that Workspace DevRel can offer some insights or even some hints, but this may indeed be infeasible.

[DemiMarie]

@junyer If it is infeasible, would that not mean that this is a bug in GMail? Hopefully Workspace DevRel can get the backend code fixed.

[junyer]

Based on the code that I have seen and also my notes, my understanding is that Gmail munges plain text (i.e. wraps lines and replaces line terminators) for compliance with RFC 2822, that it does so whenever a draft is sent and that sending via the API involves sending a draft created either explicitly or implicitly. Despite the implications for sendgmail, I don't imagine that I can argue that there's a bug per se. All we can really hope for here is a dependable workaround that isn't, you know, using SMTP.

[favonia]

@DemiMarie @junyer My personal opinion is that maybe we should blame Git more for using a practically unreliable method. That is, git send-email should probably come up with a standardized way to send patches as attachments using MIME. No email server will change a bit of an attached file, and thus we will have a reliable workflow. The reason I'm blaming Git more is that Git patches are very sensitive to even a change of a space character, so the format is arguably not "plain text". That is, it is very structured and is arguably closer to a special binary file that uses printable characters for markups. Even if someone manages to argue that it is "plain text", related RFCs arguably never emphasized that plain text should be kept intact byte-to-byte. In fact, many editors may re-wrap things as they wish, and you cannot really blame Gmail for re-wrapping things if you view the Gmail API as a way to "edit an email and send it as in the web interface" not a way to "send an email as it is directly". (You could blame Gmail for not using some cooler soft line breaks etc., but that's probably not a "bug".) Some email implementations happen not to touch "plain text" at all and thus meet git send-email's high expectations, but I feel that is a big "if" on the entire email tool-chain. It's simply more reliable to use attachments, that is, to send the files generated by git format-patch as attachments. git am can be updated accordingly to identify patches sent as attachments.

[favonia]

@junyer If anything, maybe Gmail API can provide a "RawSend" method.

[junyer]

The Linux kernel development community is... unequivocal... on the matter. ;)

FWIW, I honestly believe that the simplest thing that could possibly work here – and that would conveniently improve security across all non-API usage! – would be to support fine-grained OAuth2 scopes for IMAP, POP and SMTP. I just found this issue, BTW, and I can't say that I disagree with the sentiment expressed.

[favonia]

The Linux kernel development community is... unequivocal... on the matter. ;)

Only 95% in my opinion. They had to allow this for practicality:

Exception: If your mailer is mangling patches then someone may ask you to re-send them using MIME.

I think the practical issues mentioned in the document can be solved in another way. However, any of these is beyond my open-source social "pay grade" :laughing:

---

And yes, I agree with the sentiment in the linked issue about fine-grained OAuth2 scopes for IMAP/POP/SMTP.