Download attachment make account blocked

[PaulRichez]

When I download an attachment from a message, I can't list message from this account at the same time :

{ "statusCode": 504, "error": "Gateway Time-out", "message": "Timeout waiting for command response [T2]", "code": "Timeout" }

Futhermore, the speed downlaod look like 100kb/s, have you got an idéa why it's so slow ?

[andris9]

EmailEngine uses a single IMAP connection for each account and thus can only process a single IMAP command at a time. If you are downloading an attachment, then all other requests are queued and will be processed once the download is finished. If the download takes a long time, then the pending requests will time-out by then.

100kb/s seems slow. I guess it mostly depends on the latency and connection speed between EmailEngine and the IMAP server. There are things to tweak on the protocol side that would increase that speed, I'll keep this in mind for future EmailEngine releases.

To prevent timeouts, increase the "Max command duration" config option (see here).

Alternatively, for faster listings, it is possible to use ElasticSearch as a local cache. In that case, EmailEngine would not list messages from IMAP directly but from the local cache, so it would be faster and without limits (you can run as many parallel requests against ElasticSearch as your server could handle, unlike with IMAP where it's always one command at a time). Read more here.

[PaulRichez]

I will look on elastic search, thx for the suggestion

[PaulRichez]

Hummm it is possible to change it for get multiple IMAP connection for an account for get multiple command on same time ? For example, we can't download at the same time multiple attachment , its a problem for me

[andris9]

Unfortunately, no. EmailEngine's architecture currently does not allow more than a single connection open at any time. In general, you can't reliably open more than 2-3 simultaneous IMAP connections anyway. There are obviously providers that are fine with many connections but the big ones (Gmail, Yahoo, etc) have a really small available pool of connections available for each account. You should probably consider some kind of pre-caching where you download available attachments already when not asked for but when usage for that account is low.

The next release will get some options to tweak download speed. The reason for slow speed is chunking - EmailEngine does not download the entire attachment at once as it would take a lot of RAM. Instead, it downloads smaller chunks and sends these to the client in order. If the latency between the client, EmailEngine, and the IMAP server is high, then this adds a lot of overhead delay time for each such chunk, making the progress slow for larger attachments with a lot of chunks. The next release allows increasing the chunk size, so that EmailEngine would make fewer requests to get the attachment.

[andris9]

The new v2.23.0 version allows changing the download chunk size which makes IMAP downloads faster (at the expense of using more memory). You can set it with EENGINE_CHUNK_SIZE environment variable. The default size is 64kB.

$ export EENGINE_CHUNK_SIZE=2000000
$ emailengine

[bluepuma77]

@andris9 I would expect people sending >25k emails/month prefer to run their own SMTP/MTA server like mailu, instead of using Gmail and Yahoo. With an own local mail server it would potentially make sense for emailengine to use parallel IMAP connections to increase speed, for example of saving emails to sent folder.

[andris9]

@bluepuma77 if sending large volume of emails (you can do this by using the SMTP gateway feature in EmailEngine) it is not a good idea to upload these to the Sent Mail folder. Even if using parallel IMAP connections it does not help much as there can be only a few connections open (for example Gmail allows up to 15 IMAP connections for all IMAP apps of the same account). Large volume email sending is something you can do with EmailEngine but there are way better tools that you should consider using first. EmailEngine works better for automation use cases, not mass mailing.