Test Suite

[Mischback]

The Python-based test suite should verify, that the general setup of the different components is working.

Prerequisites

• reference the actual mailserver (bare-metal or container setup) as mail.sut.test providing (virtual) mail addresses @sut.test

Test Cases

• [x] SMTP (:25)

  • This is pretty straight forward: every mail is considered one test case, with an expected result.
  • [x] TODO: Now we have parsing of the virtual_alias config, can we use this as an input to the SMTP test suite, to send a mail to every valid recipient?
  • The expected result is provided in a dictionary.

  • [x] TODO: As of now, the implementation of SmtpTestSuite makes assumptions about the setup of the virtual alias database; this makes the whole test suite highly dependent on the actual test setup; this might be fixed by just returning a dict of the form

    result = {
      "mails_sent": [a, b, c, d, e, f, g],
      "mail_rejected": [e, f],
      "user_one@sut.test": [a, c],
      "user_two@sut.test": [c],
      "alias_one@sut.test": [b],
      "alias_list@sut.test": [d],
      "alias_invalid@sut.test":[g],
    }

    Then, the test runner tests/mailsrv_test_suite.py can merge the results of SmtpTestSuite and SmtpStarttlsTestSuite and process them by resolving the aliases.

  • [x] TODO: The internal knowledge of the test setup is also reflected in the suite's logging messages. This should be fixed.
  • [x] TODO: When the above two things are fixed, SmtpTestSuite may be simplified significantly!
  • [x] send mail to user_one@sut.test:
  • MAIL FROM: sender@another-host.test
  • RCPT TO: user_one@sut.test
  • message is accepted and placed in the (virtual) Maildir of user_one@sut.test
  • [x] send mail to alias_one@sut.test:
  • MAIL FROM: sender@another-host.test
  • RCPT TO: alias_one@sut.test
  • message is accepted and placed in the (virtual) Maildir of user_one@sut.test
  • [x] send mail to user_one@sut.test and user_two@sut.test:
  • MAIL FROM: sender@another-host.test
  • RCPT TO: [user_one@sut.test, user_two@sut.test]
  • message is accepted and placed in the (virtual) Maildir of user_one@sut.test and user_two@sut.test
  • [x] send mail to alias_list@sut.test:
  • MAIL FROM: sender@another-host.test
  • RCPT TO: alias_list@sut.test
  • message is accepted and placed in the (virtual) Maildir of user_one@sut.test and user_two@sut.test
  • [x] send mail to idontevenexist@sut.test:
  • MAIL FROM: sender@another-host.test
  • RCPT TO: idontevenexist@sut.test
  • message is rejected because the user does not exist!
  • [x] send mail to someone@another-host.test
  • MAIL FROM: sender@another-host.test
  • RCPT TO: someone@another-host.test
  • message is rejected, because the server (SUT) does not relay mails without authentication
  • [x] send mail to alias_invalid@sut.test
  • MAIL FROM: sender@another-host.test
  • RCPT TO: alias_invalid@sut.test
  • message is accepted, but a bounce message (non delivery notification?) is emitted
  • [x] use STARTTLS command
• [x] POP3 (:110) / POP3s (:995)
  • [x] Verify, that logins can only be performed over a secure connection (STARTTLS)
  • [x] check the mailbox for expected messages
  • [x] pick up and parse the protocol of the SMTP test suites, check all local mailboxes
• [x] SMTP (submission :587)
  • [x] Any "sending address" (might be determined from the LHS in util/test_configs/postfix_sender-login-map) should be able to send mail to local / internal accounts and external accounts
  • [x] Sending of mails does require a valid login (RHS in util/test_configs/postfix_sender-login-map)
  • [x] sending addresses may be usable by different login accounts (RHS of util/test_configs/postfix_sender-login-map might be a list); this has to be cross-checked!
  • [x] actual logins have to be performed via a secure connection (TLS/STARTTLS)

Required Infrastructure

• [x] user_one@sut.test is an actual vmail user with a password in Dovecot's userdb
• [x] user_two@sut.test is an actual vmail user with a password in Dovecot's userdb
• [x] alias_one@sut.test is a virtual alias to user_one@sut.test
• [x] alias_list@sut.test is a virtual alias to user_one@sut.test and user_two@sut.test
• [x] alias_invalid@sut.test is a virtual alias to a non existent mailbox user_non@sut.test
  • make sure user_non@sut.test is not available in virtual_mailboxes and vmail_users
• [x] idontevenexist@sut.test Make sure this account does not exist in Dovecot's userdb, Postfix's virtual_mailboxes database and Postfix's virtual_aliases database

[Mischback]

Adjust the output of the test suite.

https://stackoverflow.com/a/16974979 suggests defining own logging levels, which may cover the gap between DEBUG and INFO and https://stackoverflow.com/a/35804945 shows how to do this (final implementation: https://github.com/madphysicist/haggis/blob/master/src/haggis/logs.py)!

[Mischback]

https://rszalski.github.io/magicmethods/

[Mischback]

Colored Output of the test runner

• https://stackoverflow.com/questions/7445658/how-to-detect-if-the-console-does-support-ansi-escape-codes-in-python
• https://stackoverflow.com/questions/384076/how-can-i-color-python-logging-output
• https://medium.com/analytics-vidhya/python-logging-colorize-your-arguments-41567a754ac

[Mischback]

Note

To make the name resolution work during the test suite, the SUT's /etc/hosts is modified to include another-host.test.

For Postfix to pick this up, the following setting has to be applied

smtp_host_lookup = dns,native

By default, Postfix will only use the (system-configured) DNS server to resolve hosts, which is generally desired. Adding native enables lookups from /etc/hosts for hosts, that are not resolvable by DNS (and the *.test TLD is never resolvable by DNS).

Things to consider

• [ ] are there negative impacts of this setting?
• [ ] should it only be applied for testing or may it be included in the actual configuration?
• [ ] (there might be lots of other things)