search

jens-maus / yam

:mailbox_with_mail: YAM (short for 'Yet Another Mailer') is a MIME-compliant open-source Internet email client written for Amiga-based computer systems (AmigaOS4, AmigaOS3, MorphOS, AROS). It supports POP3, SMTP, TLSv1/SSLv3 connection security, multiple users, multiple identities, PGPv2/v5 encryption, unlimited hierarchical folders, an ARexx interface, etc...
https://yam.ch
GNU General Public License v2.0
63 stars 18 forks source link

YAM 2.8 SMTP helo contains not the FQDN #368

Closed jens-maus closed 8 years ago

jens-maus commented 8 years ago

Originally by yam.asc@schwarzes.net on 2013-01-11 15:49:26 +0100

When sending emails with YAM 2.8 the SMTP HELO contains only the hostname of the sender but not the FQDN, which is not correct and can cause problems with strict helo filtering.

With YAM2.7 the problem does not occur.

jens-maus commented 8 years ago

Originally on 2013-01-11 15:54:50 +0100

Can you please give detailed examples. In fact, YAM tries really hard since 2.8 to find out a reasonable FQDN of the client the mail is sent from and use that in the SMTP HELO phase of the SMTP connection. So we need some real-life examples were it still fails. So please catch some "YAM DEBUG" logs if you can and attach them to this ticket. Please also state which OS and TCP/IP stack you are using.

jens-maus commented 8 years ago

Originally by anonymous on 2013-01-11 17:24:39 +0100

Ups, I forgot to tell that I'm using MorphOS 3.1.

Seee the debug log below.

SERVER[0094]: 220 ozelot.schwarzes.net ESMTP Sendmail 8.14.5/8.14.5; Fri, 11 Jan 2013 16:51:38 +0100 (CET)
CLIENT[0015]: EHLO mosquito
SERVER[0091]: 250-ozelot.schwarzes.net Hello mosquito.schwarzes.net [62.109.78.35], pleased to meet you
SERVER[0025]: 250-ENHANCEDSTATUSCODES
SERVER[0016]: 250-PIPELINING
SERVER[0014]: 250-8BITMIME
SERVER[0010]: 250-SIZE
SERVER[0009]: 250-DSN
SERVER[0010]: 250-ETRN
SERVER[0043]: 250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN
SERVER[0015]: 250-DELIVERBY
SERVER[0010]: 250 HELP
CLIENT[0062]: MAIL FROM:<Andreas.Schwarz@schwarzes.net> SIZE=432 BODY=7BIT
SERVER[0056]: 250 2.1.0 <Andreas.Schwarz@schwarzes.net>... Sender ok
CLIENT[0041]: RCPT TO:<Andreas.Schwarz@schwarzes.net>
SERVER[0059]: 250 2.1.5 <Andreas.Schwarz@schwarzes.net>... Recipient ok
CLIENT[0006]: DATA
SERVER[0050]: 354 Enter mail, end with "." on a line by itself
CLIENT[0053]: From: Andreas Schwarz <Andreas.Schwarz@schwarzes.net>
CLIENT[0002]:
CLIENT[0051]: To: Andreas Schwarz <Andreas.Schwarz@schwarzes.net>
CLIENT[0002]:
CLIENT[0037]: Date: Fri, 11 Jan 2013 16:51:24 +0100
CLIENT[0002]:
CLIENT[0053]: Message-ID: <41e403f44c1.7129ed28@mail.schwarzes.net>
CLIENT[0002]:
CLIENT[0052]: User-Agent: YAM/2.8 (MorphOS; PPC; rv:20121224r6476)
CLIENT[0002]:
CLIENT[0020]: OpenPGP: id=661AB571
CLIENT[0002]:
CLIENT[0017]: Subject: Testmail
CLIENT[0002]:
CLIENT[0017]: MIME-Version: 1.0
CLIENT[0002]:
CLIENT[0024]: Content-Type: text/plain
CLIENT[0002]:
CLIENT[0002]:
CLIENT[0006]: Hello,
CLIENT[0002]:
CLIENT[0002]:
CLIENT[0002]:
CLIENT[0013]: best regards,
CLIENT[0002]:
CLIENT[0007]: Andreas
CLIENT[0002]:
CLIENT[0002]:
CLIENT[0005]:
.
SERVER[0056]: 250 2.0.0 r0BFpcaE090192 Message accepted for delivery
CLIENT[0006]: QUIT
SERVER[0051]: 221 2.0.0 ozelot.schwarzes.net closing connection
jens-maus commented 8 years ago

Originally by anonymous on 2013-01-11 17:24:57 +0100

I'm using the included IP Stack (NetStack).

jens-maus commented 8 years ago

Originally on 2013-01-11 17:32:24 +0100

Ok, thanks for the logfile. However, I cannot see any error regarding the identified hostname 'mosquito'. So where exactly is your problem?

BTW: please create yourself an account here on yam.ch because then we don't have to accept your comments everytime since they are flagged for moderation due to your anonymous posting.

jens-maus commented 8 years ago

Originally by yam.asc@schwarzes.net on 2013-01-11 17:57:44 +0100

The problem is "mosquito" is not "Fully Qualified", it should be "mosquito.schwarzes.net".

jens-maus commented 8 years ago

Originally by yam.asc@schwarzes.net on 2013-01-11 17:57:57 +0100

BTW. I'm logged in, but it's still moderated.

jens-maus commented 8 years ago

Originally on 2013-01-11 20:01:26 +0100

Replying to DrB:

The problem is "mosquito" is not "Fully Qualified", it should be "mosquito.schwarzes.net".

Well, first of all you seem to have no direct problem with the current way YAM issues the EHLO command, right? This is good :)

Nevertheless, I fully understand your concerns regarding the current way YAM issues a EHLO/HELO command to the SMTP server it connects to and the implications for potential SPAM rejection concerns. However, as you might know, there is no easy and straight forward way of finding out the FQDN of a host which sends out mail via SMTP via a mail user agent (MUA). While for servers (MTA<>MTA traffic) this is usually easier (since they have official public IPs and properly configured DNS and rDNS entries), a client machine behind a firewall or router usually has a private IP address and will just be configured to carry a hostname (if at all) and no FQDN. If now a MUA (e.g. YAM) behind such a firewall/router tries to send an email via SMTP to a server it can only put either the private IP address or the hostname in the EHLO command statement because that is the only information it can receive from the machine it is running on. There is no easy and straight forward way a local machine behind a firewall can find out which public IP address it really sends the mail from or what the full qualified domain name of this IP is. Thus, it can only put the hostname or the IP in a literal string "[X.X.X.X]" as defined in RFC 2821.

But now the good news. You don't really need to worry that your mails are suddenly being flagged as SPAM due to YAM putting "just" the hostname in the EHLO command. Why? Because SPAM checking mechanisms on mail servers are usually just doing EHLO checks in case a mail is received from a different mail server (MTA<>MTA traffic). This is the case because nowadays as an email user you are anyway required to use SMTP-AUTH mechanisms or the mail server knows your IP range as trusted, thus is not doing any EHLO checking at all.

If you want to read more in detail on that matter have a search on google for "SMTP FQDN EHLO" and you will find numeros webpages and forum entries about that matter. As a starter I would suggest reading the following page which should outline the matter and give the same conclusion as my statements above:

http://www.linuxmagic.com/best_practices/valid_helo_domain.html

Furthermore, if you want to see which "name" YAM usually puts behind the EHLO command and how it tries to retrieve a FQDN have a look at the following lines in tcp/Connection.c:

[source:trunk/src/tcp/Connection.c#L908]

Last not least, please note that the way YAM is trying to retrieve a valid string for the EHLO command is perfectly in line in how other modern mail clients are doing it.

jens-maus commented 8 years ago

Originally by yam.asc@schwarzes.net on 2013-01-12 00:59:09 +0100

Replying to damato:

Replying to DrB:

The problem is "mosquito" is not "Fully Qualified", it should be "mosquito.schwarzes.net".

Well, first of all you seem to have no direct problem with the current way YAM issues the EHLO command, right? This is good :)

It's working after disabling the specific helo/ehlo check at the MSA/MTA, before, the delivering was rejected with a 5xx. I've noticed the problem, because I was not able to sent emails.

Jan 11 15:30:40 ozelot sm-mta[89998]: r0BEUe5r089998: Milter: helo=mosquito, reject=554 5.7.1 Malformed HELO (not a domain, no dot)

But now the good news. You don't really need to worry that your mails are suddenly being flagged as SPAM due to YAM putting "just" the hostname in the EHLO command. Why? Because SPAM checking mechanisms on mail servers are usually just doing EHLO checks in case a mail is received from a different mail server (MTA<>MTA traffic). This is the case because nowadays as an email user you are anyway required to use SMTP-AUTH mechanisms or the mail server knows your IP range as trusted, thus is not doing any EHLO checking at all.

Thank you for the explanation about your idea how the world is working, but in the real world you will find many specific MSA/MTA installations which are also valid (and some of then are checking such things (with or without SMTP-AUTH in MUA<->MSA context)).

Last not least, please note that the way YAM is trying to retrieve a valid string for the EHLO command is perfectly in line in how other modern mail clients are doing it.

There are ~5 different clients which using this MSA/MTA, but only yam 2.8 is not able to format a proper helo/ehlo line. I don't know was yam 2.7 (and all versions before) did, but there was no problem. I'll reinstall yam 2.7 and check when I'm back at home.

jens-maus commented 8 years ago

Originally on 2013-01-12 09:02:27 +0100

Replying to DrB:

Replying to damato:

Well, first of all you seem to have no direct problem with the current way YAM issues the EHLO command, right? This is good :)

It's working after disabling the specific helo/ehlo check at the MSA/MTA, before, the delivering was rejected with a 5xx. I've noticed the problem, because I was not able to sent emails.

Jan 11 15:30:40 ozelot sm-mta[89998]: r0BEUe5r089998: Milter: helo=mosquito, reject=554 5.7.1 Malformed HELO (not a domain, no dot)

Well, obviously the server in question is controlled by you. In fact, I am also administrating a sendmail server and I know that you usually have to explicitly enable the EHLO checking as it is not enabled by default.

But now the good news. You don't really need to worry that your mails are suddenly being flagged as SPAM due to YAM putting "just" the hostname in the EHLO command. Why? Because SPAM checking mechanisms on mail servers are usually just doing EHLO checks in case a mail is received from a different mail server (MTA<>MTA traffic). This is the case because nowadays as an email user you are anyway required to use SMTP-AUTH mechanisms or the mail server knows your IP range as trusted, thus is not doing any EHLO checking at all.

Thank you for the explanation about your idea how the world is working, but in the real world you will find many specific MSA/MTA installations which are also valid (and some of then are checking such things (with or without SMTP-AUTH in MUA<->MSA context)).

I am not saying the way YAM is deciding what to present at the EHLO command is perfect. And I would be more than happy if you could assist in finding the optimum. I was just explaining the reasonings and the complications when trying to finding out a correct FQDN of the host YAM is running. And the numerous discussions on the net show that not only YAM is fighting with that issue. Nevertheless, the issue is not really that problematic since most MTA really only check the EHLO command when unauthenticared traffic is due.

Last not least, please note that the way YAM is trying to retrieve a valid string for the EHLO command is perfectly in line in how other modern mail clients are doing it.

There are ~5 different clients which using this MSA/MTA, but only yam 2.8 is not able to format a proper helo/ehlo line. I don't know was yam 2.7 (and all versions before) did, but there was no problem. I'll reinstall yam 2.7 and check when I'm back at home.

You don't need to reinstall 2.7 since I can tell you what is the difference. In 2.7 the user had to specify a valid domain himself which YAM then put behind the EHLO command. As said, since 2.8 we are trying to automatically find out a valid string to put behind the EHLO.

As said, I would be happy if you vould assist in finding a new optimum. So if you are running 5 different clients, can you please find out what they are using for the FQDN? That would really help in having s better picture of the variations used. Thanks in advance.

jens-maus commented 8 years ago

Originally by yam.asc@schwarzes.net on 2013-01-12 18:20:44 +0100

Replying to damato:

Well, obviously the server in question is controlled by you. In fact, I am also administrating a sendmail server and I know that you usually have to explicitly enable the EHLO checking as it is not enabled by default.

Yes you are right. I'm forced to enable several milters (spamass, greylist, etc.), because I'm (and my family, some friends) using the main MX domain since 15 years for usenet, different forums, shops, etc., in the peak times (2005-2008) I got over 2000 spam mails a day (maybe you know these problems). I've not changed (excepting updates) the setup since 5 years. You are also right when you say that ehlo checking is not needed for mua<->mta context, but it was enabled and causes no problems.

I am not saying the way YAM is deciding what to present at the EHLO command is perfect. And I would be more than happy if you could assist in finding the optimum. I was just explaining the reasonings and the complications when trying to finding out a correct FQDN of the host YAM is running. And the numerous discussions on the net show that not only YAM is fighting with that issue. Nevertheless, the issue is not really that problematic since most MTA really only check the EHLO command when unauthenticared traffic is due.

I think the easiest way for a fix, is to add again a configuration field for the domain (like yam2.7). It's surely a little bit problematic to detect the fqdn automatically. Maybe you can try to resolve the own IP address, or take the domain part of the smtp server. Both are only hacks, I understand that you don't want to implement this.

You don't need to reinstall 2.7 since I can tell you what is the difference. In 2.7 the user had to specify a valid domain himself which YAM then put behind the EHLO command. As said, since 2.8 we are trying to automatically find out a valid string to put behind the EHLO.

As said, I would be happy if you vould assist in finding a new optimum. So if you are running 5 different clients, can you please find out what they are using for the FQDN? That would really help in having s better picture of the variations used. Thanks in advance.

I'm using only two clients, there are others which are using the mailserver, I'll ask them.

My second client is sylpheed (3.3.0), very nice client (opensource), like yam, using it on the notebook when I'm on travel. Sylpheed has a special config option (domainname) for this.

jens-maus commented 8 years ago

Originally on 2013-01-12 21:50:13 +0100

Replying to DrB:

Replying to damato:

Well, obviously the server in question is controlled by you. In fact, I am also administrating a sendmail server and I know that you usually have to explicitly enable the EHLO checking as it is not enabled by default.

Yes you are right. I'm forced to enable several milters (spamass, greylist, etc.), because I'm (and my family, some friends) using the main MX domain since 15 years for usenet, different forums, shops, etc., in the peak times (2005-2008) I got over 2000 spam mails a day (maybe you know these problems). I've not changed (excepting updates) the setup since 5 years. You are also right when you say that ehlo checking is not needed for mua<->mta context, but it was enabled and causes no problems.

Interestingly, I am running a similar setup (sendmail+milters) since over ten years now and Inhave also played with numeros different milters and methods to keep SPAM traffic low. Out of my experience the following methods are the only sensible ones and are efficiently fighting SPAM:

  1. spamassassin+spamass: the most efficient way to prevent tons of SPAM to be accepted. However, spamassassin have to be tunes itself to carry a global autowhitelist and bayes filter throught all user accounts. In adding pyzor, razor and ddc should be enabled with spamassasin.
  2. DNS blacklisting: the second most efficient way to block SPAM as these blacklists are well maintained.
  3. DKIM/SPF signing: to reduce the probability that mails from your domain will be flagged as SPAM themselves it is a good idea to implement DKIM and SPF mechanisms in you domain. Many SPAM engines are considering DKIM/SPF setups today and will consider valid DKIM/SPF mails to be flagged as Ham.
  4. while greylisting was very efficient years ago (> 5 years) most of the SPAMers have already adapted they engines by today. Was also using greylisting in the beginning but nowaday itis not really efficient anymore and just delays your transfers. There are numeros webpages talking about this fact already. In my setup I replaced greylisting with DNS blacklisting and ended up with higher SPAM recognition rates and less false-positives.
  5. All users sending mail through such a sendmail server using a MUA have to use SMTP-AUTH with SSL. I have therefore disabled plaintext auth and direct sending of emails without authentication. In addition, sendmail have to be configured to skip all SPAM related milters iin case. Not only allows such SMTP AUTH to disable less efficient SPAM checks like the EHLO checks, but other mail servers are also sometimes recognizing the authentication headers added by such a server and potentially rating such mails as Ham as well.

I am not saying the way YAM is deciding what to present at the EHLO command is perfect. And I would be more than happy if you could assist in finding the optimum. I was just explaining the reasonings and the complications when trying to finding out a correct FQDN of the host YAM is running. And the numerous discussions on the net show that not only YAM is fighting with that issue. Nevertheless, the issue is not really that problematic since most MTA really only check the EHLO command when unauthenticared traffic is due.

I think the easiest way for a fix, is to add again a configuration field for the domain (like yam2.7). It's surely a little bit problematic to detect the fqdn automatically. Maybe you can try to resolve the own IP address, or take the domain part of the smtp server. Both are only hacks, I understand that you don't want to implement this.

Well, I removed the configuration option because in most other clients such an optin is not really available and the potentials are too high that an incorrect domain name is used by a user. Therefore I replaced the manual option with an automatic methods which (as it seems) is still not optimal. To improve the situation I have the the following in mind:

  1. change the FQDN function to use a full domain name which adds the mail server address domain if only a hostname without a domain padt was found.
  2. add a hidden option to allow users to specify their FQDN so that the function will use that one instead.

You don't need to reinstall 2.7 since I can tell you what is the difference. In 2.7 the user had to specify a valid domain himself which YAM then put behind the EHLO command. As said, since 2.8 we are trying to automatically find out a valid string to put behind the EHLO.

As said, I would be happy if you vould assist in finding a new optimum. So if you are running 5 different clients, can you please find out what they are using for the FQDN? That would really help in having s better picture of the variations used. Thanks in advance.

I'm using only two clients, there are others which are using the mailserver, I'll ask them.

My second client is sylpheed (3.3.0), very nice client (opensource), like yam, using it on the notebook when I'm on travel. Sylpheed has a special config option (domainname) for this.

Ok. Well then I have to do some more research on that matterr myself and see how other clients deal with the situation. I did that in the past already when I added the FQDN function but another search on the net might be a good idea.

jens-maus commented 8 years ago

Originally by yam.asc@schwarzes.net on 2013-01-13 16:59:20 +0100

Replying to damato:

Replying to DrB:

Interestingly, I am running a similar setup (sendmail+milters) since over ten years now and Inhave also played with numeros different milters and methods to keep SPAM traffic low. Out of my experience the following methods are the only sensible ones and are efficiently fighting SPAM:

  1. spamassassin+spamass: the most efficient way to prevent tons of SPAM to be accepted. However, spamassassin have to be tunes itself to carry a global autowhitelist and bayes filter throught all user accounts. In adding pyzor, razor and ddc should be enabled with spamassasin.

Nearly the same setup here spamass-milter with addauth patch, spamassassin + razor.

  1. DNS blacklisting: the second most efficient way to block SPAM as these blacklists are well maintained.

I've dropped this, because this is also scored by spamassasin (several dnsbl).

  1. DKIM/SPF signing: to reduce the probability that mails from your domain will be flagged as SPAM themselves it is a good idea to implement DKIM and SPF mechanisms in you domain. Many SPAM engines are considering DKIM/SPF setups today and will consider valid DKIM/SPF mails to be flagged as Ham.

Yes, I'm using SPF to protect my domain (old TXT RR). For incoming mails I've used spf-milter, but this ist also covered by spamassasin, so there is no need to use the extra milter.

  1. while greylisting was very efficient years ago (> 5 years) most of the SPAMers have already adapted they engines by today. Was also using greylisting in the beginning but nowaday itis not really efficient anymore and just delays your transfers. There are numeros webpages talking about this fact already. In my setup I replaced greylisting with DNS blacklisting and ended up with higher SPAM recognition rates and less false-positives.

There is still one advantage, until the greylisting time is expired, the spammer run in several spam traps, which increas the dnsbl scoring.

  1. All users sending mail through such a sendmail server using a MUA have to use SMTP-AUTH with SSL. I have therefore disabled plaintext auth and direct sending of emails without authentication. In addition, sendmail have to be configured to skip all SPAM related milters iin case. Not only allows such SMTP AUTH to disable less efficient SPAM checks like the EHLO checks, but other mail servers are also sometimes recognizing the authentication headers added by such a server and potentially rating such mails as Ham as well.

Unfortunately not all of the users are using SMTP AUTH, there is also some "smtp after pop" hack installed, or they are local (like me). There are also some scripts which want to send email in certain situations. Sure, it's possible to change this, but this needs time.

I think the easiest way for a fix, is to add again a configuration field for the domain (like yam2.7). It's surely a little bit problematic to detect the fqdn automatically. Maybe you can try to resolve the own IP address, or take the domain part of the smtp server. Both are only hacks, I understand that you don't want to implement this.

Well, I removed the configuration option because in most other clients such an optin is not really available and the potentials are too high that an incorrect domain name is used by a user. Therefore I replaced the manual option with an automatic methods which (as it seems) is still not optimal. To improve the situation I have the the following in mind:

  1. change the FQDN function to use a full domain name which adds the mail server address domain if only a hostname without a domain padt was found.
  2. add a hidden option to allow users to specify their FQDN so that the function will use that one instead.

Both solutions would be fine for me.

jens-maus commented 8 years ago

Originally on 2013-01-19 01:32:49 +0100

In (f8885a2):

jens-maus commented 8 years ago

Originally on 2013-07-02 21:26:16 +0200

Release Note: fixed a problem where under certain setups the automatically chosen full qualified domain name (FQDN) was unluckily chosen and thus mail sending being rejected when using YAM. Now YAM should use a better mechanism and even let you override the FQDN using a new hidden 'MachineFQDN' option.