[MU] [DRAFT] Mailet to handle mailing lists

chibenwa commented 3 months ago

This ticket needs to be validated with @guimard

What

We want to write a mailet implementing basic mailing list features on top of a LDAP.

How

Step 1: group expension

(We just implement the openList)

We define hereby a pattern for mailing list naming:

listname@lists.domain.tld

Where domain.tld is a domain managed by the backend (in the domain list) and lists.domain.tld is a domain managed too.

After validating the format and existance of the domains (here lists.domain.tld and domain.tld),

We search (LDAP search) from the mail address the distinguish name (dn) of the list. Here:

dn of the group: cn=listname,ou=lists, dc=domain,dc=tld

Based on this dn, retrieves the group members (attribute memberOf) and substitute the group (removed from recipients) by the list of members mail address.

Step 2: Caching

Use cafeine in memory cache to save user dn => mailaddress correspondance.

The cache will drop least read entries, with a ttl of 24h, and define a maximum count of entries, configurable via mailet properties.

This cache will significantly decrease LDAP calls, only needed to get DN of members (1 call) and not to get the user DNs (N calls).

This DN => mail caching can be extracted into a dedicated component, binded in Guice as a singleton.

Step 3: sender restrictions

Why? Some lists are private and only list members have the right to write to it.

What? We define several sender validation policy, encoded as part of the businessCategory:

openList: no sender validation is performed, anyone can write to the list
internalList: only local users (that have a mail address on the James server) can write to the list. External emails are rejected.
memberRestrictedList: only members of the list can write to the list.
ownerRestrictedList: only ownlers of the list can write to the list. owners typically are a subset of members
domainRestrictedList: only accept sender with the same domain name than the list. This typically is a subset of local users (some local users may have different domains and thus will be prohibited to write to the list.

How? After retrieving a group,

Based on the businessCategory choose the senderValidation policy, and feed it the group LDIFF in order to validate if the sender is allowed or not.

If sender is allowed , rewrite the list normally.

If sender is denied, remove the group from the recipients, and send a copy of the email (MAIL FROM: sender, RCPT TO: list) to the rejectedSender processor (configurable through the rejectedSenderProcessor property).

Step 5 List headers

Add the following per-recipient headers for list members:

List-Post: <mailto:sales@lists.linagora.com>
List-Id: <sales@lists.linagora.com>

Evolutions

(NOT to be implemented for MU governement)

OpenRegistration: This kind of list allows users to enroll them selves by writing to a subscribe address, and similary unsubscribe.

Encoding: this can be done by adding the -openRegistration suffix to the business category. This gives for instance the following valid businessCategory: openList-openRegistration, internalList-openRegistration, domainRestrictedList-openRegistration.

In case of OpenRegistration, we would need to handle mails to: mylist-subscribe@lists.domain.tld and mylist-unsubscribe@lists.domain.tld and add/remove members onto the group on the LDAP accordingly. Note that new added members needs to match the sender validation policy.

List-Unsubscribe header needs to be positionned onto a per user basis for list members:

List-Unsubscribe: <mailto:news-unsubscribe@lists.linagora.com>

Please note that the act of subscribing might be spoofed and used to overflow with mails a given mail address. Thus there needs to be a validation process:

Define a salt in the configuration
Derive a validation token using SHA-256 using the mail address of the new member, the salt and the name of the list.
Send a message: hello, we well received a request to subscribe to XyZ mailing list, in order to subscribe please answer to this mail (loaded from a template for I8N) - the validation token is then passed onto the reply-to mail address: Reply-To: <news-unsubscribe+8f3fg3ifg232hf2if3i47fidibcwie@lists.linagora.com>
- Upon receiving the second reply the list will validate the token. If valid list member is added. If invalid the mail is dropped.

Appendix

LDIFF data proposal

# List open to potential external customers, contact point of the company
cn=sales,ou=lists,dc=linagora,dc=com
mail: sales@lists.linagora.com
businessCategory: openList
member: uid=hansolo,dc=linagora,dc=com
member: uid=leila,dc=linagora,dc=com

# List open to employees to ask questions about Java to java experts
cn=internal-java-support,ou=lists,dc=linagora,dc=com
mail: internal-java-support@lists.linagora.com
businessCategory: internalList
member: uid=btellier,dc=linagora,dc=com
member: uid=rcordier,dc=linagora,dc=com

# Private list reserverved solely to Cosoft members
cn=cosoft,ou=lists,dc=linagora,dc=com
mail: cosoft@lists.linagora.com
businessCategory: memberRestrictedList
member: uid=btellier,dc=linagora,dc=com
member: uid=xguimard,dc=linagora,dc=com

# This list allows mailing all the staff and thus is restricted
cn=staff,ou=lists,dc=linagora,dc=com
mail: staff@lists.linagora.com
businessCategory: ownerRestrictedList
owner: uid=vsteffen,dc=linagora,dc=com
owner: uid=azapolsky,dc=linagora,dc=com
owner: uid=mmaudet,dc=linagora,dc=com
member: uid=btellier,dc=linagora,dc=com
member: uid=mmaudet,dc=linagora,dc=com
member: uid=azapolsky,dc=linagora,dc=com
member: uid=vsteffen,dc=linagora,dc=com
member: uid=xguimard,dc=linagora,dc=com
member: ...

# This list is only for vn office members with @vn.linagora.com addresses
cn=vnoffice,ou=lists,dc=vn,dc=linagora,dc=com
mail: vnoffice@lists.vn.linagora.com
businessCategory: domainRestrictedList
member: uid=vttran,dc=vn,dc=linagora,dc=com
member: uid=hphan,dc=vn,dc=linagora,dc=com
member: uid=hqtrong,dc=vn,dc=linagora,dc=com
member: ...

Sample configuration

Put it into mailetcontainer.xml > transport BEFORE the RRT mailet

<mailet matcher="All" class="TMailMailingLists">
    <rejectedSenderProcessor>rejectedSender</rejectedSenderProcessor>

    <!-- hexdump -vn16 -e'4/4 "%08X" 1 "\n"' /dev/urandom -->
    <salt>D5EC6B0CDEC4732061D919C5EE9B2BAF</salt>

    <userAddressCacheMaxEntries>20000</userAddressCacheMaxEntries>
    <userAddressCacheTTL>1day</userAddressCacheTTL>

    <groupCacheMaxEntries>20000</groupCacheMaxEntries>
    <groupCacheTTL>1day</groupCacheTTL>
</mailet>

chibenwa commented 3 months ago

https://github.com/chibenwa/openpaas-james/tree/ldap-mailing-lists

Implemented steps 1, 2, 3, 5.

Now remains the hardest: testing!

chibenwa commented 3 months ago

https://github.com/linagora/tmail-backend/pull/1135 tests added, the resulting code shall be pretty conclusive...

chibenwa commented 3 months ago

Known limitations & upcoming work on this topic

This 1 weekend coding party was great but thinking about it there's several rough corners still ahead.

[x] Interactions with RRTs (forwards, aliases, etc...)

The current implementation decorelates group handling from RRT

This come at a price: with the current proposed pipeline, if a user forward emails to a list, then the list will not be resolved (as the list mailet would be positionned before the RRT).

Of interest this situation can be solved by patching the local-address-error:

We could implement a IsALDAPList matcher
And a ToProcessor -> root mailet associated with IsALDAPList matcher within local-address-error processor.

This is a rare occurrence so a bit of ceremony and extra LDAP requests might be acceptable here.

[X] Loop prevention with RRT

 GIVEN bob is a member of **mygroup@lists.linagora.com**
 AND bob forwards his mails to **mygroup@lists.linagora.com**
 WHEN bob receives an email
 THEN we shall not create an infinite loop

This means that groups shall be integrated in the loopPrevention local mechanism.

[x] Interaction with SMTP RCPT validation

The groups are currently NOT considered a local address. This means that ValidRCPTHandler would reject the recipients, thus not allowing SMTP users / MXs to send email to the group.

We would need to implement an addition TMailListAwareValidRCPTHandler that would take the current heuristic for lists (lists prefix, local address...) and a cache (as we do not want to lookup LDAP for each and every recipient) - or a periodical refresh from a LDAP list is maybe even better... A few minutes delays after group creation might we acceptable...

We could also ship a configurable mode that accept emails for all local domains starting with lists. While this would accept too many mails this would clearly be less costly in terms of performance and only cost a few Cassandra queries.

[ ] This proves the importance of some integration tests exercising the SMTP layer, and the forwards.
[x] Ideally we shall use a single LDAP-POOL. We can refactor James ReadOnlyUsersRepository in order to extract the repository instanciation. And directy inject the LDAP pool in all Mailet/matcher related to the LDAP.

Mutualizing the LDAP connections will be important to scale well in our usage of LDAP.

chibenwa commented 3 months ago

To be noted that groupOfNames LDAP objectClass do not have mail attribute.

In order to do so, specific schema needs to be crafted.

This means that we should make it easy to change the objectClass used to define groups, might it be needed.

This also means that, in order to ease testing we can assume the description attribute holds the mail. As long as the attribute is configurable, it would be easy to adapt TMail lists to the final schema. This workaround was validated with @guimard .

BTW here is the link to the LDIF, for review of non-james + LDAP specialists DEVs

https://github.com/chibenwa/openpaas-james/blob/ldap-mailing-lists/tmail-backend/mailets/src/test/resources/ldif-files/populate.ldif

chibenwa commented 3 months ago

PR for mutualizing the LDAP connection pool in James: https://github.com/apache/james-project/pull/2359

chibenwa commented 3 months ago

I merged the pull request for the time being.

linagora / tmail-backend