Emails don't seem to roundtrip

NorfairKing commented 2 years ago

I don't know if the RFC's fault, or the implementation, but it looks like parsing and rendering messages doesn't roundtrip.

roundtrip

Is this intentional?

frasertweedale commented 2 years ago

Thanks for reporting. Which release / commit?

frasertweedale commented 2 years ago

Nevermind, I see the problem. The problem is that the content-type header is not represented in the construction, and as a result, the message is not recognised as multipart when parsing. The upcoming release v0.5.0 should address this (and breaks the API by changing the data types). Do you want to try testing with master branch?

NorfairKing commented 2 years ago

@frasertweedale I cannot upgrade because you require random >=1.2 and I'm on lts-16.12...

NorfairKing commented 2 years ago

It would also be nice if roundtrip tests like this were part of the test suite. Now I have to write tests to find bugs in the library rather than in my own. I'd be happy to help with writing tests if you like.

frasertweedale commented 2 years ago

I have message roundtrip tests in the test suite. See https://github.com/purebred-mua/purebred-email/blob/master/tests/Message.hs#L73-L95. The failure you encountered does not arise in my tests because the generator uses the "proper" APIs for building multipart messages, and these added the required headers.

Generating all possible constructions of the MIMEMessage data type with v0.4.3 will yield values that don't round trip (as you have seen). It will be less of an issue in v0.5. But it will still be possible to create messages that don't round trip, for example by explicitly adding a Content-Type header that disagrees with the content-type information in the dedicated constructor fields.

IMF and MIME are complex. Although I believe it is possible to define data structures that can only represent conformant messages, and which round trip perfectly, the complexity would be enormous. I would prefer to focus on ergonomic interfaces for common use cases than go down that very deep rabbit hole.

frasertweedale commented 2 years ago

@NorfairKing Of course, if you want to help develop purebred-email, your contributions are most welcome! But perhaps we can start with a question: what is your use case, and what are the gaps and pain points in purebred-email for your use case?

NorfairKing commented 2 years ago

The failure you encountered does not arise in my tests because the generator uses the "proper" APIs for building multipart messages, and these added the required headers.

AHA! We may want to add some comments about that. Not every value is valid (that's what validity is for). I will try this and report back. Please excuse my lack of knowledge about how MIME works.

IMF and MIME are complex. Although I believe it is possible to define data structures that can only represent conformant messages, and which round trip perfectly, the complexity would be enormous. I would prefer to focus on ergonomic interfaces for common use cases than go down that very deep rabbit hole.

Yes I think that's entirely right. Thanks for pointing this out and not jumping down that rabit hole for me!

@NorfairKing Of course, if you want to help develop purebred-email, your contributions are most welcome! But perhaps we can start with a question: what is your use case, and what are the gaps and pain points in purebred-email for your use case?

I'm considering building a haskell email client here: https://github.com/NorfairKing/kalm Nothing works yet, but I've made a diagram: diagram

On the topic of random >=1.2, do you need it? It would be really nice if we could relax that constraint.

frasertweedale commented 2 years ago

@NorfairKing are you aware of https://github.com/purebred-mua/purebred? It's a terminal MUA using notmuch for mail storage, and sits in pretty much the same box as Kalm in that diagram above. We wrote purebred-email for working with the message data.

I don't wish to deter you from doing your own thing - but if the goals are similar and you like purebred's approach (mutt-esque UX) it could be good to work together.

For sure, I can add comments about validity (I think I already did in the unreleased code on master, but could probably say more). I will consider Validity type class, and thank you for bringing it to my attention.

frasertweedale commented 2 years ago

As for random >= 1.2, I am using the new and much improved types and classes that landed in 1.2. I'm not keen to revert it because some of those types also appear in the purebred-email API, for random generation of MIME things that need uniqueness (multipart boundaries, message IDs, etc).

NorfairKing commented 2 years ago

@frasertweedale I wasn't aware of purebred, I thought purebread-email was a reference to the fact that it was written in pure Haskell. This is a super cool project and it would be amazing if I don't have to write my own.

I have a few concerns, that are probably work-around-able:

I would want to integrate sending and receiving into the project, not just the reading, organising and queuing of messages.
A piece of tooling this critical to my workflow will require extra careful review from myself, and that's something that would be easier if I had written/contributed to the code
I vehemently disagree with some parts of the HACKING (but very much agree with the rest):
- Use the weakest abstraction possible:
- Optics are your friend
- No stack build
Anyway, it's too harsh of me to provide feedback like this but I felt it important to bring up when you talk about considering working together.

frasertweedale commented 2 years ago

@NorfairKing all good. At the end of the day if we can't agree, we can do different things. But it is good if we can find common ground and work from there. It might be better to discuss each of the topics in more detail in issues filed against purebred, but here are some brief thoughts:

stack build

I have no interest personally, but no fundamental objection. We used to have it at the beginning, but it made testing very difficult due to "the xmonad way" of configuration - that is, recompilation (see https://hackage.haskell.org/package/dyre for details). So we removed it. If you want to own it (with or without tests), you can have it. Just keep in mind we are sometimes using versions of libraries that are not in Stackage or LTS releases.

sending

It's a general interface, you could implement an SMTP sender today. OOTB we ship an implementation that invokes a local mailer. We use a different implementation in the test suite to put "sent" mail aside for inspection.

An SMTP mailer should be provided as a separate "add-on" library, not part of core purebred (to avoid the dependencies).

receiving

Could you describe in more detail your requirements? With purebred, mail is stored on the filesystem and indexed using notmuch. We designed purebred with an assumption that an external MDA (e.g. fetchmail + maildrop) will take care of retrieval/delivery. The problem is well solved and we didn't see any reason to solve it again using Haskell. That said, I do see some possible advantages of integrating mail retrieval/delivery with purebred:

fewer components for users to configure (at the expense of more stuff to configure in purebred itself)
better "new mail" notification UX
better ways to pre-process mail, e.g. for automatic tagging, with access to the full power of Haskell and purebred-email for analysing received messages

If you really need it, we will design and develop the interfaces to solve it, and the implementation(s) can be shipped as add-on libraries.

abstractions and optics

Difference of opinion ;)

frasertweedale commented 2 years ago

I'm closing this now that v0.5 was released, which contained some of the type improvements discussed above.

purebred-mua / purebred-email