Integration test suite

[ivy]

As it stands now, there are only unit tests that mock the behavior but don't actually interact with an SMTP server. I believe we can merge with more confidence if we can start testing against a locally running server.

[pedromorgan]

Maybe this https://github.com/mailhog/MailHog

[ivy]

No, this needs to be a real MTA or we risk missing important differences in widely deployed implementations. MailHog looks like a helpful UI for local development.

[marcusva]

Docker comes to mind with something like the following setup:

• one container containing go-mail including a test application sending out 1-N mails covering go-mail's feature set
• one container running an SMTP server receiving mails and storing them into a mbox- or maildir-based inbox

The test script would start up both containers e.g. via docker-compose, wait for the mails to be sent and stored (simple approach: wait X seconds) and compares the stored mails against the expected set of mails.

The latter would need to be created beforehand, though, with some important points to consider:

• timestamp mismatches in the headers
• sender/receiver information (IP) mismatches
• other header mismatches

The comparison might be done using the mbox format (two files to compare) with the necessity to filter out variable parts, as mentioned before.

A more complex approach might be to parse the stored mails and just evaluate certain portions, such as

• From, To, Bcc, Sender
• Subject
• Body
• multipart content
• other go-mail configurable message settings

against the predefined set of information. This might be accomplished with reasonable effort using e.g. Python's email package (or third-party libraries).

The advantage of the second approach is that the input data could be kept external and used for both, feeding the mail producer as well as the comparison script. This also would reduce the maintenance efforts for the test scenarios.

Since docker can be run within a Travis-CI environment, this also lets one test pull requests and other code changes against the integration test suite without any additional effort.

[pedromorgan]

@marcusva I got a couple of virtuals on machines here so docker would be neat. One runs postfix, one is google proxy config, and a souple of exchange server (spits) etc.. mainly to test our clients systems..

[pedromorgan]

it all runs on fabfile script at the moment.. but checking against real machines in a way..

[james-lawrence]

@ivy on a related note, library should expose the current mocking functionality (maybe from a subpackage) we've run into a bug were partial emails are delivered in the original library but no way to really reproduce without copying a bunch of code from the library.

[ivy]

@james-lawrence Interesting, were you able to find out the cause? Do you think better debug logging would have helped in your case?

[james-lawrence]

yes, the cause is assets for the email being unavialable during the creation of the email and the email contents are written directly to the transport layer. so if anything goes wrong a partial email is sent.

[james-lawrence]

debug logging is immaterial here, it was more an issue with writing a test to prevent regressions.

when I said reproduce, I mean in tests, not in the actual system.

[phamdt]

@ivy i'd be curious to know what mailhog is lacking for testing. i'm using it myself for testing features that use go-mail and want to make sure there's not some sort of case that needs more attention.