Ah, what a journey this PR has been... Let me explain everything.
My first step in trying to fix the email issue on Tower was reaching out to Seqera Labs and pinging the Nextflow Slacks. Sadly, they were not super helpful, mainly because Seqera said "this works on Tower on our end." and Nextflow Slacks didn't respond. It happens.
On Tower, the issue seems like sendmail or mail utils are not present on the base AWS Batch ECS image. I could probably create my own AMI, but that seems like an unnecessary source of later technical debt.
My initial suspicion was that our SMTP secrets were not getting through. In order to prevent that, I hardcoded those values to make sure they were getting through. That was not an issue, but I realized for certain SMTP providers, you also have to input a FROM address, because otherwise Nextflow defaults to the instance address. Along the way, I also noticed a typo with SMTP vs SMPT. All fixed now.
Seems like the existing sendmail and mail implementation that comes with the nf-core templates is quite old. And there are manymanymany people struggle to get this working.
Turns out, Nextflow actually has its own sendMail utility and it's been an age-long issue to switch to using that in nf-core.
Sadly, it was almost impossible to find any examples of people making this work using Nextflow's sendMail. This is most likely because the existing nf-core version worked decently well on HPCs (many comments about this). But the new built-in method is supposed to work across the spectrum. I thought this would be a win! So I decided to go at it alone.
This led me to spelunk into Nextflow's repository to understand how their built-in sendMail works. That was difficult to fully figure out, because
I didn't know in which contexts I could call built-in methods (turns out it works in onComplete but doesn't in any of the classes under lib.
I didn't know what the right arguments were due to poor documentation. There are many places with conflicting instructions as to specifying the body of the email (body vs content, first is correct) and the attachments (attach vs attachment vs attachments, first one is correct).
Once I figured out everything, I tested it all end-to-end on an EC2 instance and it all seems to work. I need to do further testing on Tower, so please don't merge yet. As a nice side benefit of this, I got to delete bunch of template code!
Finally, I've attempted some stuff that didn't work out on Tower (like using Nextflow's tempFiles. The commit history of this PR explains it all.
Ah, what a journey this PR has been... Let me explain everything.
My first step in trying to fix the email issue on Tower was reaching out to Seqera Labs and pinging the Nextflow Slacks. Sadly, they were not super helpful, mainly because Seqera said "this works on Tower on our end." and Nextflow Slacks didn't respond. It happens.
sendmail
ormail
utils are not present on the base AWS Batch ECS image. I could probably create my own AMI, but that seems like an unnecessary source of later technical debt.My initial suspicion was that our SMTP secrets were not getting through. In order to prevent that, I hardcoded those values to make sure they were getting through. That was not an issue, but I realized for certain SMTP providers, you also have to input a
FROM
address, because otherwise Nextflow defaults to the instance address. Along the way, I also noticed a typo with SMTP vs SMPT. All fixed now.Seems like the existing
sendmail
andmail
implementation that comes with the nf-core templates is quite old. And there are many many many people struggle to get this working.Turns out, Nextflow actually has its own
sendMail
utility and it's been an age-long issue to switch to using that in nf-core.Sadly, it was almost impossible to find any examples of people making this work using Nextflow's
sendMail
. This is most likely because the existing nf-core version worked decently well on HPCs (many comments about this). But the new built-in method is supposed to work across the spectrum. I thought this would be a win! So I decided to go at it alone.This led me to spelunk into Nextflow's repository to understand how their built-in
sendMail
works. That was difficult to fully figure out, becauseonComplete
but doesn't in any of the classes underlib
.body
vscontent
, first is correct) and the attachments (attach
vsattachment
vsattachments
, first one is correct).Once I figured out everything, I tested it all end-to-end on an EC2 instance and it all seems to work. I need to do further testing on Tower, so please don't merge yet. As a nice side benefit of this, I got to delete bunch of template code!
Finally, I've attempted some stuff that didn't work out on Tower (like using Nextflow's
tempFile
s. The commit history of this PR explains it all.