search

stalwartlabs / jmap-server

Stalwart JMAP server
https://stalw.art/jmap
GNU Affero General Public License v3.0
616 stars 14 forks source link

Object storage for attachments #9

Closed sstubbs closed 1 year ago

sstubbs commented 2 years ago

Hi,

firstly thanks for this amazing project. I’m interested in trying this out. Currently I’m using https://maddy.email which works well but generally most of my stack is rust so would prefer to use this for the imap storage.

Using jmap as the foundation then an imap proxy is an absolutely brilliant idea. I feel like this project could be a game changer.

I understand the data is stored in rocksdb. Is it possible to store the attachments on object storage like minio?

If so this truly would check all the boxes for my use.

mdecimus commented 2 years ago

Hi,

Adding support for MinIO is fairly simple as it was partially implemented but then removed from the first release. The reason why it was removed is that it was not as performant as the local storage. Additionally, Stalwart JMAP includes native clustering support and all objects can be easily replicated by adding more nodes.

That being said, I will leave this issue open as it won't hurt having the option to store objects on MinIO/S3 if the administrator prioritizes using a third party object storage over having a slightly slower performance accessing blobs.

BTW, how did you find this project? I'm curious because it won't be announced until September 21st :-)

sstubbs commented 2 years ago

Oh interesting. I found this as I was actually thinking of perhaps starting a project like this as I’ve run email servers over the last decade and they’ve always been a pain. So did a bit of digging on GitHub.

Really excited to have found this. Happy to contribute where I can.

When I realised jmap was getting added to cyrus a few years ago I was really happy but then scaling things to do with jmap does’t work the same way as their standard mailboxes so ended up ditching jmap and going back to code that interfaces with imap. Cyrus itself is a bit of a painful task config wise generally.

For my personal mailboxes I’ve been testing out maddy email.

It works well backed by yugabytedb and minio but I know there are locking issues and wouldn’t use it with anything else.

Currently I use nvme drives for yugabytedb for databases in the servers and ssds for minio. So ideally I would like to have this project storing its rocksdb database on nvme drive on each server using your clustering solution and attachments stored on minio.

when prices come down on nvme drives I would happily just use this for everything but ideally would like the choice as attachments can use quite a lot of storage.

The prospect of having this project for all email data and surrealdb as my only database for app data one day in the future is very exciting thought. :)

sstubbs commented 2 years ago

I would be deploying this on kubernetes by creating a helm chart and probably an operator at some point in the future using kube.rs if I need it for admin tasks. So I could potentially share these unless this is something already on your roadmap to do internally.

mdecimus commented 2 years ago

when prices come down on nvme drives I would happily just use this for everything but ideally would like the choice as attachments can use quite a lot of storage.

Makes sense. Just wanted to clarify that the raw messages are stored on the blob storage, not just the attachments. The blob storage is only accessed with the user wants to access the contents of an e-mail message, for everything else the metadata stored in RocksDB is used.

I would be deploying this on kubernetes by creating a helm chart and probably an operator at some point in the future using kube.rs if I need it for admin tasks. So I could potentially share these unless this is something already on your roadmap to do internally.

Deploying on Kubernetes is on the roadmap, but before that we need to write the Jepsen tests so the clustering mode can leave the beta. In any case, if you want to start testing the server on kubernetes, that would be great!

sstubbs commented 2 years ago

Ok I understand. Makes sense. For my use the full raw message in blob storage is great as that means even less on the nvme drives. Minio is fairly quick so I think it wouldn’t slow down viewing message content to an unusable level.

I’ll start testing this out.

batistein commented 1 year ago

@sstubbs did you already deployed this project to kubernetes? I'm right now testing this out and came across this issue. Maybe it makes sense to join forces if you already created an helm chart for example.

mdecimus commented 1 year ago

Stalwart JMAP v0.3 now supports an S3-compatible store.