search

mail-in-a-box / mailinabox

Mail-in-a-Box helps individuals take back control of their email by defining a one-click, easy-to-deploy SMTP+everything else server: a mail server in a box.
https://mailinabox.email/
Creative Commons Zero v1.0 Universal
13.77k stars 1.43k forks source link

Forwarded mails rejected due to SPF issue #1946

Closed soletan closed 3 years ago

soletan commented 3 years ago

This is a follow-up to #1198. Solution there was to use roundcube filters for handling mail forwarding, however this doesn't work.

Use Case

All mail sent to MIAB-account (me@foo.com) should be forwarded to existing mail account in different domain (me@bar.com).

Remarks

Scenario

Expected Result

Actual Result

Attempts To Solve

Did some research on this issue. Found related forum posts. Found #1198. Stumbled over this (happens to be in German). Basically, tried first sieve script example there in slightly reduced form:

require ["fileinto", "editheader", "variables", "regex", "envelope", "copy"];
# rule:[all to target]
if envelope :matches "From" "*" {
    deleteheader "From";
    deleteheader "To";
    addheader "From" "me@foo.com";
    addheader "Reply-To" "${1}";
    addheader "To" "me@bar.com";
    redirect "me@bar.com";
}

This script has been rejected due to missing plugin editheader (at least).

In addition, I don't think this is a feasible solution for regular users lacking in-depth knowledge about what is going on in mail delivery or even in this script.

jvolkenant commented 3 years ago

I used to be able to get by with using an alias to forward email@mydomain.com to gmail.com for my wife, but after more people started using DMARC, gmail stopped playing nice with forwarded mail.

The way I have to do it now is just let mail land in the inbox and have gmail pop pull the mail that way it's not being forwarded.

Is something like that available for "me@bar.com"?

soletan commented 3 years ago

Well, fetching mails is a feasible workaround, but no solution to this issue. Same applies to simply requiring every interested user to set up another account in every software on every device used to check for mails.

Just consider having set up MIAB for a school with teachers and pupils as users. I can't expect everyone to know their privately used MTA and whether fetching mails is supported there. I can't ask them to set up another account, nor can I ask them to paste a custom sieve script in their RC webmailer. At least I can't do so without negative impact on this particular tool's overall acceptance.

Or consider an administrator who decides to support different parties in managing their MIAB instances. It would be eligible to have an account on either MIAB instance, e.g. for accessing admin views or for testing bugs reported by either party. I don't consider it reasonable to set up time-based fetchmail for every instance causing lots of useless traffic every now and then to poll either instance's account for new system notifications or mails sent by one of its users.

Tight SPF checks are quite common nowadays. IMHO there is no solution unless there is a way for a regular non-techie user to set up a simple forwarding for an actual account (which is still required to access the additional features of MIAB).

Lecquio commented 3 years ago

I think SRS is needed for this to work

myfirstnameispaul commented 3 years ago

@soletan

It seems like your use-case is outside the goals of the Mail-in-a-Box project, which was originally created as a personal or possibly SOHO mail server that is self-configuring. It is nearly impossible to provide a self-configuring mail server that can meet all use cases for a project as small as MiaB. The examples you are providing are cases that seem like there should either be an on-staff IT person to configure a suitable solution or an outside service provider.

Perhaps a different project can already perform you desired function, such as iRedMail, Modoboa, mailcow, etc., as they have more features related to overall mail and mail server administration.

That said, I don't mean to discourage working out custom configurations, as many here do (including myself), nor dissuade you from submitting a PR if you feel a discovered solution will make the project better (the creator of the project often encourages such).

JoshData commented 3 years ago

I'm pretty sure Roundcube filters used to forward mail correctly (i.e. passes SPF), so it would be helpful if anyone else can confirm the issue and if it's confirmed figure out what changed that might have broken this.

soletan commented 3 years ago

@myfirstnameispaul can't argue with that ... though the scenarios might apply to SOHO setups as well.

soletan commented 3 years ago

Just to clarify: the MTA receiving the forwarded mail is seeing the original sender's address in SMTP MAIL FROM command (envelope address). So, from that MTA's point of view a mail is delivered by MIAB but originating from a domain this MIAB isn't authoritative for.

myfirstnameispaul commented 3 years ago

This is the expected behavior for an alias created in the MiaB dashboard, which is why it has a warning that even veteran users often overlook.

It is not clear to me as well why the Roundcube solution isn't working for you. Many other users have reported this resolves their issue.

soletan commented 3 years ago

But I'm not using an alias. And if so, an alias might be rendering itself rather useless as well. If target MTA is applying strict SPF checks that behaviour is violating SPF.

soletan commented 3 years ago

AFAICS this setting is essential here: sender_redirect_envelope_from. I haven't found it in any file of a 0.52 MIAB installation, so I assume the default applies. Maybe, orig_recipient would be a better fit, though. I have to admit, that I'm not quite as familiar with Sieve in general or with Pigeonhole in particular as I am with general MTA setup, thus my question: are those settings controllable from a sieve script by means of setting a variable or do they work in system configuration files, only?

myfirstnameispaul commented 3 years ago

Is it possible you could provide a more detailed description of what you have tried, what isn't working, and what your desired result is?

I just did the following test:

Sent an email from username@yahoo.com to username@example.com, where example.com is hosted by MiaB. In Roundcube I configured a rule requiring from:username@yahoo.com and to:username@example.com redirect to username@gmail.com. Now when I send an email from username@yahoo.com to username@example.com it is automatically forwarded to username@gmail.com. Gmail reports the SPF check is neutral and DKIM and DMARC pass for yahoo.com. (Note the emails went to the Spam folder but I suspect this is typical oligopolistic behavior due to Google likely having not seen mails coming from example.com before.)

Unless I am missing something, this seems like it conforms to your desired result.

soletan commented 3 years ago

Gmail reports the SPF check is neutral

I guess this is the essential delta. In my case the sender domain is explicitly providing an asserted list of valid senders. The MIAB instance isn't part of it, thus SPF actually fails rather than causing neutral result. Having SPF neutral due to ?all seems feasible for large-scale MTAs like Yahoo, but due to those large-scale hosters as well as third-party scoring services like to bully smaller MTA providers I prefer anything but neutral SPF for my domains and MIAB users can't control all the probable sender domains they are receiving mails from.

myfirstnameispaul commented 3 years ago

So I did the test in reverse and the mail inboxes fine with Yahoo!.

Header checks report spf=softfail smtp.mailfrom=gmail.com. DKIM and DMARC both report as passed.

I experienced this with both the 'redirect' and 'send message copy to' options in the filter, so I'm not clear on what the difference is between those two options. Note I don't currently have a domain to send from not hosted by MiaB that uses hardfail.

If the only acceptable solution is to pass SPF checks, then you likely need to evaluate the relatively new options employed by mailing lists and figure out how to implement that.

soletan commented 3 years ago

If the only acceptable solution is to pass SPF checks, then you likely need to evaluate the relatively new options employed by mailing lists and figure out how to implement that.

I'm more and more confused. And it looks like I don't know what you are referring to here. This issue is about SPF fail. So, if a remote MTA is stubborn enough to keep enforcing SPF, MIAB is failing to deliver mail there by design and any MIAB user considering this to be an issue with MIAB should evaluate operation of separate additional tools as a workaround?

Apart from that, this whole discussion makes me wonder how to read this statement found on MIAB homepage (which was one of the major reasons for me to promote MIAB in the first place):

Note that the goals of this project are to . . .

  • [..]
  • Promote decentralization, innovation, and privacy on the web.

Concluding posts so far, I would state: MIAB plays nicely as a junior partner to central MTA providers and so this issue isn't a bug in MIAB, but some personal feature request of mine. All it takes to prove that is showing a working setup with differently operating peers?

Currently it feels like to be either user's fault to interact with peers at custom MTAs who happen to share interest in decentralizing services rather than limiting communications to those people satisfied with using GMail and Yahoo.

And I still don't see any MIAB user - techie or non-techie - being able to enforce certain SPF records provided by every mail senders' domain administrator nor to assure that any available Forward-MTA is relaxing its SPF checking usually in use as a counter measure to UBE.

FWIW, I used to test "redirect" previously, only. When using "redirect" failed forwards are hard to track back to the sender so those failures might stay unnotified by either party. But I tried "send copy to" as an option now. Even that one is failing due to SPF fail (not softfail) at Forward-MTA. As a minimum benefit I get a failure notice from MIAB MTA, though:

Recipient address rejected: Message rejected due to: SPF fail - not authorized. Please see http://www.openspf.net/Why?s=mfrom;id=me@bar.com;ip=1.2.3.4;r=<UNKNOWN> (in reply to RCPT TO command)

I have to admit, that the failure appears in RCPT-TO-command rather than MAIL-FROM-command as stated before. But this isn't affecting the reason, just the time of checking SPF on incoming mails. BTW, 1.2.3.4 is a replacement, but is meant to represent the MIAB instance's public IP.

soletan commented 3 years ago

Tried to adopt fix I've suggested before. I've adjusted file /etc/dovecot/conf.d/99-local by appending this rule at end of existing plugin section:

sieve_redirect_envelope_from = orig_recipient

and restarted dovecot afterwards. Next time mail forwarding was working as expected for this time the envelope from address was me@foo.com. Checking mail in target the original sender and forwarding recipient were still available, so a user would be able to realize, that this mail has passed MIAB instance and would be able to respond to original sender by replying as usual.

myfirstnameispaul commented 3 years ago

I'm more and more confused. And it looks like I don't know what you are referring to here. This issue is about SPF fail. So, if a remote MTA is stubborn enough to keep enforcing SPF, MIAB is failing to deliver mail there by design and any MIAB user considering this to be an issue with MIAB should evaluate operation of separate additional tools as a workaround?

I am only referencing your specific issue, which is not the normal use. The normal use case is users sending and receiving mail with user accounts on MiaB using their various different client devices or the webmail interface.

Since MiaB is not a part of any large mail provider and is a relatively easy mail server to configure, users can easily migrate away from the large platforms of Google, Microsoft, Yahoo!, etc., thus promoting decentralization.

Other mail server options are substantially more work to configure and maintain. To achieve the simplicity of having a self-configuring server, MiaB cannot offer every single feature for every use-case, even when the installed software can be configured to support far more use-cases.

As far as your specific issue, which is what I was trying to address in previous posts, you have the same problem as other mail forwarding services, such as mailing lists (e.g., Sympa, Mailman, etc.) and you may be able to determine a customized configuration of MiaB that supports your use-case.

One project that is working to resolve this issue is ARC, which could be implemented in MiaB if someone took the time to work out a configuration and submit a PR.

That said, if the receiving servers you are sending to refuse to accept anything other than what is stated in an SPF record, and they don't conform to any other standards, such as the newer ARC standard, then the only thing you can do is send them emails IAW the SPF record as there is no MTA configuration to fix an unauthorized IP address issue.

myfirstnameispaul commented 3 years ago

It's possible I've missed something:

Do you mean that you want the email to be from the MiaB hosted domain address, even though it is originating from a non-MiaB hosted domain email address?

soletan commented 3 years ago

It's possible I've missed something:

Do you mean that you want the email to be from the MiaB hosted domain address, even though it is originating from a non-MiaB hosted domain email address?

As far as I am aware, that is the issue triggering tight SPF checks at target MTAs on forwarding mails, yes. I want envelope-from to show the actual situation: MIAB is sending a mail to another MTA, not some MTA, which was sending that mail to MIAB before that.

soletan commented 3 years ago

That said, if the receiving servers you are sending to refuse to accept anything other than what is stated in an SPF record, and they don't conform to any other standards, such as the newer ARC standard, then the only thing you can do is send them emails IAW the SPF record as there is no MTA configuration to fix an unauthorized IP address issue.

In this you are wrong. Those MTAs don't refuse to accept what is stated in SPF, but fully acknowledge it. The SPF simply disables any unlisted host from sending mails on behalf of the SPF record's domain. Yahoo and Google are different in that, causing neutral and softfail results. But these are just two out of several more options SPF records may state. And it is to the opposite, that MIAB is currently failing to consider those MTAs that actually accept all kinds of SPF records when it comes to forwarding mails through sieve filtering.

myfirstnameispaul commented 3 years ago

In this you are wrong. Those MTAs don't refuse to accept what is stated in SPF, but fully acknowledge it. The SPF simply disables any unlisted host from sending mails on behalf of the SPF record's domain. Yahoo and Google are different in that, causing neutral and softfail results. But these are just two out of several more options SPF records may state. And it is to the opposite, that MIAB is currently failing to consider those MTAs that actually accept all kinds of SPF records when it comes to forwarding mails through sieve filtering.

On this we have some sort of a miscommunication. Here is what I am stating:

$ dig txt example.com +short
"v=spf1 ip4:203.0.113.81 -all"

With this record the only way to pass the SPF check is for the smtp.mailfrom=example.com to come from the IP address203.0.113.81.

So you are stating there is an MTA configuration where the smtp.mailfrom=example.com can come from some other IP address and pass the SPF check?

soletan commented 3 years ago

No. It is not about some misconfigured MTA passing mails though SPF is requiring them to reject it. But let's stay with this example.

What if MIAB IP is not 203.0.113.81 and you are sending a mail via 203.0.113.81 with sender foo@example.com to a MIAB host? My point is: when setting up sieve in RC to forward this mail to another MTA, this MTA might check SPF and decide to reject the mail because SPF check is failing since the MIAB host is currently using foo@example.com as envelope from address provided in smtp.mailfrom. That MTA gets smtp.mailfrom=example.com because of a kind of misconfiguration in MIAB, or to be more precise and as it turns out for me, in dovecot/pigeonhole.

JoshData commented 3 years ago

Hi. Project owner here. Definitive reply follows. :)

I want envelope-from to show the actual situation: MIAB is sending a mail to another MTA

I am pretty sure (as I said previously) that this was what the sieve rules used to do. If they aren't doing that now, and if there is a simple fix to getting back to that behavior, I would gladly accept a PR to fix it.

The only thing that needs to be happening in this thread is 1) confirming the bug, 2) searching for a simple fix, and 3) figuring out why/when this behavior changed. I think @soletan has already found the fix in sieve_redirect_envelope_from so hopefully this is solvable!

soletan commented 3 years ago

Ok, tried to find a place for this setting to be applied. Found the setup/mail-dovecot.sh script using tools/editconf.py, however this time the setting isn't top level dovecot option, so it isn't as easy as copy-n-paste any of the existing lines using tools/editconf.py. Are there any equivalent cases existing in MIAB trying to set a scoped option in key-value resembling nginx configuration syntax (which accidentally happened to be a no-brainer in regard to their scopes in related setup script setup/web.sh at least)?