lieer + git send-email creates "broken" threads.

[granquet]

Hello.

Just trying out lieer 1.3 and I'm wondering if anyone is using 'gmi send' with git send-email and has encountered issues with threads? email clients such as mutt or alot see the first mail as "standalone" and the following messages correctly grouped by thread.

The In-Reply-To and References in the subsequent mails are correct and point to the first message id.

Though, lieer complains that it cannot find the parent message when sending the subsequent mails:

warning: could not find parent message, sent message will not be associated in the same thread

I tried wrapping gmi send with gmi sync && notmuch new to force an update of the database and maybe help in finding the parent message but to no avail.

I'll keep digging and update here if I have something worth sharing.

Example output:

granquet@seychelles ~/projects/conf
 % git send-email -4
/tmp/R9XUsEApXW/0001-vim-fix-pathogen-and-bundles.patch
/tmp/R9XUsEApXW/0002-Add-install-script.patch
/tmp/R9XUsEApXW/0003-vim-Make-vim-autoload-symlink-relative.patch
/tmp/R9XUsEApXW/0004-tmux-bind-C-a-a-to-send-C-a.patch
To whom should the emails be sent (if anyone)? granquet@baylibre.com
Message-ID to be used as In-Reply-To for the first email (if any)?
(mbox) Adding cc: Guillaume Ranquet <ranquet.guillaume@gmail.com> from line 'From: Guillaume Ranquet <ranquet.guillaume@gmail.com>'
(body) Adding cc: Guillaume Ranquet <ranquet.guillaume@gmail.com> from line 'Signed-off-by: Guillaume Ranquet <ranquet.guillaume@gmail.com>'
path: ~/.mail/granquet@baylibre.com
sending message, from: Guillaume Ranquet <granquet@baylibre.com>..
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  4.18it/s]
receiving metadata: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  3.63it/s]
message sent successfully: 1795be5d07c16444
OK. Log says:
Sendmail: /usr/bin/gmi send -C ~/.mail/granquet@baylibre.com -i granquet@baylibre.com ranquet.guillaume@gmail.com
From: Guillaume Ranquet <granquet@baylibre.com>
To: granquet@baylibre.com
Cc: Guillaume Ranquet <ranquet.guillaume@gmail.com>
Subject: [PATCH 1/4] vim: fix pathogen and bundles
Date: Tue, 11 May 2021 16:47:20 +0200
Message-Id: <20210511144723.32678-1-granquet@baylibre.com>
X-Mailer: git-send-email 2.26.3
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

Result: OK
(mbox) Adding cc: Guillaume Ranquet <ranquet.guillaume@gmail.com> from line 'From: Guillaume Ranquet <ranquet.guillaume@gmail.com>'
(body) Adding cc: Guillaume Ranquet <ranquet.guillaume@gmail.com> from line 'Signed-off-by: Guillaume Ranquet <ranquet.guillaume@gmail.com>'
path: ~/.mail/granquet@baylibre.com
sending message, from: Guillaume Ranquet <granquet@baylibre.com>..
looking for original message: 20210511144723.32678-1-granquet@baylibre.com
warning: could not find parent message, sent message will not be associated in the same thread
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  4.43it/s]
receiving metadata: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  3.47it/s]
message sent successfully: 1795be5d95e9bd0b
OK. Log says:
Sendmail: /usr/bin/gmi send -C ~/.mail/granquet@baylibre.com -i granquet@baylibre.com ranquet.guillaume@gmail.com
From: Guillaume Ranquet <granquet@baylibre.com>
To: granquet@baylibre.com
Cc: Guillaume Ranquet <ranquet.guillaume@gmail.com>
Subject: [PATCH 2/4] Add install script
Date: Tue, 11 May 2021 16:47:21 +0200
Message-Id: <20210511144723.32678-2-granquet@baylibre.com>
X-Mailer: git-send-email 2.26.3
In-Reply-To: <20210511144723.32678-1-granquet@baylibre.com>
References: <20210511144723.32678-1-granquet@baylibre.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

gitconfig:

[sendemail]
        smtpserver = /usr/bin/gmi
        smtpserveroption = "send"
        smtpserveroption = "-C"
        smtpserveroption = "~/.mail/granquet@baylibre.com"
        suppresscc = self
        chainreplyto = false
        composeencoding = UTF-8
        transferencoding = 8bit

[granquet]

I spent a bit more time digging into this. Although the In-Reply-To field of the subsequent messages mentions the Message-Id of the first message:

First message:
Message-Id: <20210511144723.32678-1-granquet@baylibre.com>
Subsequent messages:
In-Reply-To: <20210511144723.32678-1-granquet@baylibre.com>

The Message-Id of the first mail is rewrote by google into something like:

Message-Id: <CABnWg9tHjDu800FVqnH4nj5to2JTCOtDa4A24v4ZN4U9M06DDQ@mail.gmail.com>

Which, obviously, breaks the git sendmail thread feature.

some people seem to complain here and there that this is indeed happening to them when using the gmail API... but I couldn't find a reliable source of information on this "feature".

Not sure there's an easy fix in lieer... not sure there's a fix at all?

[gauteh]

Think a fix would require the In-Reply-To of the following messages to be updated with the new id. Seems like a lot of hassle? We could override it in gmi, but it would probably still be difficult to make git-send-email able to pass that information to gmi.

[granquet]

Think a fix would require the In-Reply-To of the following messages to be updated with the new id. Seems like a lot of hassle? We could override it in gmi, but it would probably still be difficult to make git-send-email able to pass that information to gmi.

Yeah, I do not see an easy way to fix it... unless google stops behaving likes this, I guess git-send-email + gmi is a no-go.

[mjg]

I just checked both with msmtp and with gmi send, both with my gmail address and with an address which configured as a sender address in GMail.

The result seems to be that no matter what, sending with smtp preserves message-IDs and sending via the API creates new ones.

So, in the process of ditching my app-specific passwords, I'll rather teach msmtp oauth. I'm wondering if it makes sense to teach lieer's remote.py to share the tokens so that there is only one config (using my own client secret, of course), rather than scripting around google's oauth.py for msmtp.

[gauteh]

It looks like this might be related to the format of the message-id:

• https://stackoverflow.com/questions/52013699/gmail-api-is-overwriting-the-custom-message-id-header-while-sending-emails
• https://groups.google.com/g/comp.mail.sendmail/c/gkRdBwka4iw- -> https://groups.google.com/g/comp.mail.sendmail/c/gkRdBwka4iw-/m/7DxqaxSzDwAJ

maybe git-send-email can be configured to do this?

[mjg]

My message-id's are of that form (as produced by alot, e.g. <163447991646.76807.13691714014724242560.foo@domain.com>). I'm afraid GMail recreates MIDs on "receive" when they are invalid; on "send by API" unconditionally; on "send by SMTP" probably when they are invalid (haven't checked).

The API returns the MID, but at that point the sent message is not in the notmuch db yet (only after sync). So, gmi could either:

• store a mapping original MID -> GMail MID somewhere or
• copy MID to a header X-Original-Message-ID before sending (and tell users to index that in notmuch) and then (on user's request) rewrite In-Reply-To headers before sending if they match a stored original header.

Or hope for GMail to change its behaviour :)

[gauteh]

Gmi adds the message (with the new id) to the db when it is sent, if that helps.

[mjg]

Gmi adds the message (with the new id) to the db when it is sent, if that helps.

Ah, sure, I completely missed that get_content() does that. So it adds the message as returned from the API. At that point send() still has the original e-mail object eml, can compare the MIDs and could store the original MID e.g. as a property in the notmuch db so that, in turn, it is available if send() fails to find a MID when it parses In-Reply-To headers.

I'm still wondering whether we have a confirmed example of a message sent through the API with unchanged MID, i.e. whether there is a simple way to "conform".

[gauteh]

søn. 17. okt. 2021, 16:45 skrev Michael J Gruber @.***>:

Gmi adds the message (with the new id) to the db when it is sent, if that helps.

Ah, sure, I completely missed that get_content() does that. So it adds the message as returned from the API. At that point send() still has the original e-mail object eml, can compare the MIDs and could store the original MID e.g. as a property in the notmuch db so that, in turn, it is available if send() fails to find a MID when it parses In-Reply-To headers.

Yes, or also printed out. The following messages need their in reply to modified.

I'm still wondering whether we have a confirmed example of a message sent through the API with unchanged MID, i.e. whether there is a simple way to "co

From the comments on the stack overflow Q's it seems like it is not possible when using the API.