[enhancement]: Store emails in DB

[BonGoo]

Which feature or improvement would you like to request?

I'd like to see option to store email's not only in blob (S3 like systems) but in Database (Foundation DB or SurrealDB). For my opinion it will be great option to do more scalable systems. Databases give us more flexibility, of clustering, HA, and error resiliency

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[mdecimus]

FoundationDB does not support keys larger than 100kb. It would require splitting the message in dozens or hundreds or chunks in order to do that, which will have a large performance impact.

[BonGoo]

Those two DB systems that i was type, have been only suggestion. There is so much good DB systems, SurrealDB, MongoDB, MariaDB. All of those system is fast, scalable, clustered, and modern. Maybe this system there will be good for this option.

[Andy-2639]

With MySQL 5, there are also various limits: The wire protocol limits the message size which means that one single record cannot exceed this size (I think typically 1 MB). Also, using string functions to read one cell in parts is limited as afaik the whole cell has to be loaded into memory to get the slice. So the max string length applies.

For a hobby playground project, I store files in a MySQL, now MariaDB, database and I chose to store the files in 64 KB chunks to prevent problems. At the time I made this decision, I did research about storing large blobs in MySQL. Performance isn't a problem for that project. (Neither for my stalwart-mail installation but I'm definitely not representable for stalwart-mail users.)

Reason for using a DB and not the filesystem was that I planned to allow registered users to upload files and I hesitate to use the filesystem to keep the attack vector low (path traversal attacks and such). A nice side effect is getting consistent backups for free: just use a database transaction.

Downside: filesystem seems to be much more performant than this database stuff.

[itsezc]

With #23 in mind, it is entirely possible to store Emails in the case of SurrealDB, not only would this be possible but also allow you to perform (potentially) spam filteration from ML with "learning fields" down the line, which are yet to be implemented.

Another obvious benefit is when serving large user groups a database scales easily over a filesystem, especially where unique access control and functionality is necessary, i.e. a full text search over 100s of thousands of emails

[BonGoo]

If SurrealDB will handle it. There will be great enhancement if @mdecimus will create support for this. Fingers crossed, and hopefully we will see this functionality.

[tomhaerter]

Postgres allows to store 1 GB in TEXT columns. In general, Postgres is a solid database (though it may not be as horizontally scalable out of the box as e.g. FoundationDB). Are there cases where an email message is larger than 1 GB (with attachments stored elsewhere, of course)?

EDIT: Attachments can be stored in e.g. MinIO and should never be stored in the database itself.

[BonGoo]

Postgres allows to store 1 GB in TEXT columns. In general, Postgres is a solid database (though it may not be as horizontally scalable out of the box as e.g. FoundationDB). Are there cases where an email message is larger than 1 GB (with attachments stored elsewhere, of course)?

I think that give ability to store 1GB attachment will be something terrible to maintain. What do you thing @mdecimus

[williamdes]

Terrible yes, seems like it would even be a server filling vulnerability Most email providers seem to block emails above something like 15Mb Could emails be stored in a DB and attachements in S3/Minio/local?

[tomhaerter]

Terrible yes, seems like it would even be a server filling vulnerability Most email providers seem to block emails above something like 15Mb Could emails be stored in a DB and attachements in S3/Minio/local?

As far as I know, attachments are already stored using a blob storage, e.g. MinIO. If most email providers reject anything over 15 MB, then 1GB should be enough for the email message anyway (by email message, I mean everything except the attachments)

[BonGoo]

Wow, i see that something is happening in performace repository :D, thanks @mdecimus you are doing great job.

[mdecimus]

Stalwart Mail Server v0.5.0 has just been released including support for storing blobs in the database. For more details please read the announcement, change log, upgrading instructions and updated documentation.

If you have questions or problems please start a discussion on Github or reach us on Discord/Matrix.