message id construction enhancements

[c-alpha]

Suggestions for how the message id construction could be enhanced:

1. Allow the user name part to be empty, i.e. do not add anything to the left of the @ (not even a .). Having a constant part in there seems a waste of storage space, and reduces the entropy of the random part. If the fixed value hints to the MUA client, someone knowing a vulnerability might send me a maliciously crafted message.

2. Ability to automagically set the FQDN part according to the FQDN of the From: header as an alternative to a fixed value. I use several accounts, and would like to always use a FQDN in the message id that corresponds to the FQDN I'm sending from. I am currently using a nonsense value, which formally satisfies FQDN requirements. But again that is a fixed string, reducing the entropy of the message id.

[gauteh]

1) How would that look like, should this not contain a random part?

2) So FQDN's based on account?

You could use a plugin to do this, see: https://github.com/astroidmail/astroid-plugin-seccomp

[c-alpha]

[...] • How would that look like, should this not contain a random part?

Currently, astroid's scheme is:

   "Message-ID: <" <random-string> "." <mail.message_id_user> "@" mail.message_id_fqdn ">"

When mail.message_id_user is set to the empty string (""), the string "astroid" is inserted.

My proposal would be to either drop mail.message_id_user altogether, or when set to the empty string, do not add ".astroid" anymore, but don't add anything.

The random-string to the left of the "@" should be a random value. A total length of 256 bits would seem reasonable. Hex-encoded that would make it 32 characters long.

Together with the sender's FQDN, that should make it sufficiently unique, IMO.

• So FQDN's based on account?

Yes. When sending as "john.doe@mymail.org" the message id should be "@mymail.org", and when sending as "j.doe@company.com", the message id should be "@company.com".

Maybe add a "%1" parameter for mail.message_id_fqdn which gets replaced by the FQDN found in the From header?

You could use a plugin to do this, see: https://github.com/astroidmail/astroid-plugin-seccomp [...]

This goes in a similar direction, but falls short of meeting my criteria.

First, it uses 5 consecutive calls to the random number function. I don't know what assertions can be made about the python random number generator (RNG), but if it uses a pseudo-RNG with a constant seed, it will always produce the same sequence of random numbers. I.e. after each restart of astroid, the sequence of generated message ids could be identical in the worst case.

Second, it has three fixed dashes ("-") in the left hand side. The message id is not for human use, so any syntactic sugar is a waste of resources. Imagine how many bazillions of message ids are transmitted and processed by MTAs and MUAs worldwide every day. each bit we can save makes a difference.

[gauteh]

c-alpha writes on november 15, 2017 22:15:

[...] • How would that look like, should this not contain a random part?

Currently, astroid's scheme is:

   "Message-ID: <" <random-string> "." <mail.message_id_user> "@" mail.message_id_fqdn ">"

When mail.message_id_user is set to the empty string (""), the string "astroid" is inserted.

My proposal would be to either drop mail.message_id_user altogether, or when set to the empty string, do not add ".astroid" anymore, but don't add anything.

The random-string to the left of the "@" should be a random value. A total length of 256 bits would seem reasonable. Hex-encoded that would make it 32 characters long.

Together with the sender's FQDN, that should make it sufficiently unique, IMO.

• So FQDN's based on account?

Yes. When sending as "john.doe@mymail.org" the message id should be "@mymail.org", and when sending as "j.doe@company.com", the message id should be "@company.com".

Maybe add a "%1" parameter for mail.message_id_fqdn which gets replaced by the FQDN found in the From header?

You could use a plugin to do this, see: https://github.com/astroidmail/astroid-plugin-seccomp [...]

This goes in a similar direction, but falls short of meeting my criteria.

First, it uses 5 consecutive calls to the random number function. I don't know what assertions can be made about the python random number generator (RNG), but if it uses a pseudo-RNG with a constant seed, it will always produce the same sequence of random numbers. I.e. after each restart of astroid, the sequence of generated message ids could be identical in the worst case.

Second, it has three fixed dashes ("-") in the left hand side. The message id is not for human use, so any syntactic sugar is a waste of resources. Imagine how many bazillions of message ids are transmitted and processed by MTAs and MUAs worldwide every day. each bit we can save makes a difference.

In general I agree.

In notmuch; message-ids have a bit more meaning. You can search for them, and people often refer to them with id:....@.. (which you can also do in astroid). The same is true for git send-email, where you instantly know that this is a patch just from looking at the ID. And you often reply to previously sent patch-series using just the ID.

In these cases it useful to have a ID which says something about who it's from (so that when I see the ID I might recognize the sender without having to open the mail) - or I will know that this is reference to a patch.

Either way; it is not necessary to avoid information that is otherwise present in the e-mail (like sender). A user who wishes to use hide his identity could use a custom message_id_user.

Currently the astroid message id is by default constructed out of:

%1.%2.%3@%4

where:

%1: is unix time of message creation %2: random alphanumeric string (length 10) %3: local user name %4: local fqdn (e.g. localhost.none)

The relevant code is located in edit_message.cc:183.

Currently the User-Agent is also transmitted (though you can turn this off through a plugin). Other than that it is possible to hide the local user-name and fqdn using configuration values. This follows other MUA norm. I agree that if message_id_user == "" we should just drop that field completely. More advanced message ids should be handled through the plugin. Using both current time and a relatively short (10 chars) random string should make any ID collisions unlikely enough.

I don't think python RNG is any more problematic than C++ one, haven't looked into whether it is used incorrectly here. Either way, maybe you can submit those issues to the plugin repository? Perhaps @emdete has some thoughts as well.

[c-alpha]

Once again couldn't resist, and am I'm working on a PR for this. ;-)

I have updated the factory defaults config file to set the user_id to "astroid" (same default behaviour), and the logic in edit_message.cc to only insert ".%3" if the user_id is not empty. So the end user has the choice.

I have also updated the generation of the random part to be compliant with RFC5322, and to use a 40-character random string, which gives us 253 bit of entropy, i.e. makes any two systems generating the exact same random string all but impossible. I have removed the time part, since that is predictable, i.e. especially the verbatim UNIX time is the same for all systems, and hence increases the probability of collisions.

Now I'm working on the second commit of the PR, which will implement the setting of the message id's right hand side from the sender domain.

[gauteh]

c-alpha writes on november 16, 2017 13:11:

Once again couldn't resist, and am I'm working on a PR for this. ;-)

I have updated the factory defaults config file to set the user_id to "astroid" (same default behaviour), and the logic in edit_message.cc to only insert ".%3" if the user_id is not empty. So the end user has the choice.

I have also updated the generation of the random part to be compliant with RFC5322, and to use a 40-character random string, which gives us 253 bit of entropy, i.e. makes any two systems generating the exact same random string all but impossible. I have removed the time part, since that is predictable, i.e. especially the verbatim UNIX time is the same for all systems, and hence increases the probability of collisions.

Now I'm working on the second commit of the PR, which will implement the setting of the message id's right hand side from the sender domain.

Sounds good!

[c-alpha]

new branch: c-alpha/astroid/message-id

[c-alpha]

On a related note: it doesn't seem to be possible to include wiki updates in a PR? https://github.com/astroidmail/astroid/wiki/Customizing-editor would need some small updates if and when this would get merged.

[gauteh]

c-alpha writes on november 16, 2017 18:23:

On a related note: it doesn't seem to be possible to include wiki updates in a PR? https://github.com/astroidmail/astroid/wiki/Customizing-editor would need some small updates if and when this would get merged.

The wiki is connected to a separate repository. You should be able to edit it directly.

[gauteh]

c-alpha writes on november 16, 2017 18:18:

new branch: c-alpha/astroid/message-id

Thanks, could you make it into a separate PR? Then it is easier for others to test and comment on the specific changes.

[c-alpha]

Thanks, could you make it into a separate PR? Then it is easier for others to test and comment on the specific changes.

I will once I will have the second commit ready, which will deal with the right hand side of the message id (set to FQDN of the sender).

[c-alpha]

PR #438

[c-alpha]

PR was closed, so closing this, too.