search

la5nta / pat

A cross-platform Winlink client written in Go
https://getpat.io
MIT License
490 stars 86 forks source link

act as a real email agent #96

Open anarcat opened 7 years ago

anarcat commented 7 years ago

Right now, email going in and out of pat happens through the compose and read commands. Those are not standardized interface: the prompts seem (at least to me) ad-hoc and do not seem to adhere to common standards.

But there are standards for email delivery, defined in some of the oldest RFCs out there. According to the internet, pat is a mail delivery agent (MDA): it takes email from some place and delivers it into a user's mailbox (although that mailbox is a custom format in the rather obscure ~/.wl2k/mailbox folders). It also act as a mail submission agent or mail transfer agent (MSA/MTA) in that it takes a message from a user and sends it over to another server (over AX25, "telnet" or whatever ham protocol).

An overview of those concepts is documented in the email agent wikipedia page and is also covered by RFC5598. This diagram from section 4 includes most agents in common configurations:


                     ++========++
                     ||        ||                             +-------+
          ...........++  aMUA  ||<............................+ Disp  |
          .          ||        ||                             +-------+
          .          ++=+==+===++                                 ^
          .  local,imap}|  |{smtp,submission                      .
          .  +-----+    |  |                          +--------+  .
          .  | aMS |<---+  | ........................>| Return |  .
          .  +-----+       | .                        +--------+  .
          .                | .    *****************       ^       .
          .          +-----V-.----*------------+  *       .       .
          .      MSA | +-------+  *   +------+ |  *       .       .
          .          | | aMSA  +-(S)->| hMSA | |  *       .       .
          .          | +-------+  *   +--+---+ |  *       .       .
          V          +------------*------+-----+  *       .       .
    //==========\\                *      V {smtp  *       .       .
    || MESSAGE  ||                *   +------+    *  //===+===\\  .
    ||----------||            MHS *   | MTA  |    *  ||  dsn  ||  .
    || ENVELOPE ||                *   +--+---+    *  \\=======//  .
    ||  smtp    ||                *      V {smtp  *     ^   ^     .
    || CONTENT  ||                *   +------+    *     .   . //==+==\\
    ||  imf     ||                *   | MTA  +....*......   . || mdn ||
    ||  mime    ||                *   +--+---+    *         . \\=====//
    \\==========//                * smtp}| {local *         .     ^
          .           MDA         *      | {lmtp  *         .     .
          .      +----------------+------V-----+  *         .     .
          .      | +----------+   *   +------+ |  *         .     .
          .      | |          |   *   |      | +..*..........     .
          .      | |   rMDA   |<-(D)--+ hMDA | |  *               .
          .      | |          |   *   |      | |<.*........       .
          .      | +-+------+-+   *   +------+ |  *       .       .
          .      +------+---------*------------+  *       .       .
          .  smtp,local}|         *****************       .       .
          .             V                                 .       .
          .          +-----+                         //===+===\\  .
          .          | rMS |                         || sieve ||  .
          .          +--+--+                         \\=======//  .
          .             |{imap,pop,local                  ^       .
          .             V                                 .       .
          .       ++==========++                          .       .
          .       ||          ||                          .       .
          .......>||   rMUA   ++...........................       .
                  ||          ++...................................
                  ++==========++

    Legend: --- lines indicate primary (possibly indirect)
                transfers or roles
            === boxes indicate data objects
            ... lines indicate supporting transfers or roles
            *** lines indicate aggregated service

                     Figure 5: Protocols and Services

That's an awfully complicated diagram, but it covers basically all use cases in internet email delivery: the above is an extremely complex and complete system - we don't need to implement all of that stuff. We don't even need to implement a full SMTP client or server: we just need to be able to talk to some of them. And of course, Pat doesn't speak SMTP at all right now, but it would be great if pat could integrate with existing infrastructure and thus allow email delivery to and from the normal internet. And I don't think it would be that hard to implement, quite the contrary.

For example, the "traditional" way to send email on UNIX systems is by piping the message through the sendmail command: then the message ends up in the local MTA message queue for delivery. Local delivery is a little trickier: it varies from one software to the next or, more accurately, from one storage medium to the next. But technically, the same principle could be used: just shove messages over the closest MTA and let it handle delivery. This would require some configuration to determine which email address to forward the local emails to, but I don't think that would cause much problems, other than possible mail loop if the user does something stupid. And of course, the message needs to be formatted in a specific way, with From, To, Date and Subject headers.

So, concretely, what I propose is a way for pat to pipe email messages in (MTA) and out (MDA). There should be a pat sendmail command that would allow users to pipe messages into winlink (MTA), which could then easily be used by another MTA to route some or all SMTP traffic through Winlink. For example, in the popular Postfix MTA, that would be some configuration in master.cf plus some optional pattern matching to decide which emails to route over Winlink. The other side would be some configuration that would allow the user to set a mailbox (say a Maildir folder) where emails are stored. The Maildir folder format is easy enough that it could be implemented by just slightly modifying the current routines (or building a similar one on top). Alternatively, one should be able to configure a command to pipe email messages into for delivery, and just reuse the same email generator used for the MTA part.

There were discussions touching this in #87, but they were limited to how to implement a notification system. This would allow normal user mailbox notifications to kick in, which would completely resolve that issue. Furthermore, it would allow much more user-visible flexibility in the way pat operates: it would allow it to become a full email gateway for responsible operators that are up to it.

I understand you may be hesitant in implementing this feature, but keep in mind it wouldn't fundamentally change the way Pat works: the UI wouldn't change, nor would the way emails go in and out from the Winlink perspective. It's just about adding another way to extract and inject data into Pat, and shouldn't create too much problems, hopefully. :)

Thank you for considering this request and reading this far...

martinhpedersen commented 7 years ago

Thanks for the detailed explanation anarcat.

This has been suggested by several people the last year, and the last discussion can be seen here: https://groups.google.com/forum/#!topic/pat-users/Tl0M9PwzJXM

My main concern with this approach is the differences between the Internet Message Format (RFC 5322 and 6532) and the Winlink Message Structure. My conclusion is that it is very hard to create a good translation tool between the two formats, and it may end up being confusing for users to deal with the various ambiguities that may occur.

Further more, using the JSON API that Pat exposes or the wl2k-go packages, it should be easy enough to start a separate experimental pat-sendmail project to implement this. We could even consider merging it into Pat if there is substantial interest in the community to do so in the future. (I've already seen a python script for the MTA part using the JSON API for delivery to Pat's outbox).

In addition, this sounds very similar to paclink-unix' approach and I feel like that would be a duplication of effort since users that want this can use that application on most UNIX platforms?

To summarize;

  1. I am concerned that this would greatly increase the complexity of the project (translating between formats).
  2. It can be written as a separate application using wl2k-go packages or the JSON API.
  3. This is already possible using paclink-unix.
anarcat commented 7 years ago

On 2017-09-29 17:48:50, Martin Hebnes Pedersen wrote:

Thanks for the detailed explanation anarcat.

This has been suggested by several people the last year, and the last discussion can be seen here: https://groups.google.com/forum/#!topic/pat-users/Tl0M9PwzJXM

My main concern with this approach is the differences between the Internet Message Format (RFC 5322 and 6532) and the Winlink Message Structure. My conclusion is that it is very hard to create a good translation tool between the two formats, and it may end up being confusing for users to deal with the various ambiguities that may occur.

Further more, using the JSON API that Pat exposes or the wl2k-go packages, it should be easy enough to start a separate experimental pat-sendmail project to implement this. We could even consider merging it into Pat if there is substantial interest in the community to do so in the future. (I've already seen a python script for the MTA part using the JSON API for delivery to Pat's outbox).

Thanks for the references and detailed response (even though this was discussed on mailing lists ;).

For me, it suffices that there is some openness that this could be eventually merged into pat. it doesn't need to be done right now, or even for 1.0, if you insist (although I think it would be an important enough feature to delay 1.0 if there's energy to implement this).

Any reference for that Python script? I'm fluent in python as well and could give it a shot.

I agree converting between Winlink messages and RFC5335 (and especially RFC6532, ie. i18n) is the hard part in the project. I somehow completely missed that in my analysis, but it was in the back of my mind the whole time. :)

I'm not sure I like the JSON interface, but if it's what we get, okay. ;) I would personally favor a pat-native implementation that would support serializing data structures to different formats. RFC5335 is, after all, just one message format - you may want to save messages as static HTML for archival purposes or something... It seems to me an such an implementation would just have to implement the InboundHandler/OutboundHandler interfaces to pipe emails in and out... Then, as you say, the main challenge is (de)serialization of the Message objects, but looking at the header.go file is encouraging: there's MIMEHeader in there. ;)

In addition, this sounds very similar to paclink-unix' approach and I feel like that would be a duplication of effort since users that want this can use that application on most UNIX platforms?

i never got around to using paclink, for some reason. it looks much harder to deploy than pat.

To summarize;

  1. I am concerned that this would greatly increase the complexity of the project (translating between formats).

sure, it would increase complexity of some bits, but it could remain completely optional and out of the main code paths.

  1. It can be written as a separate application using wl2k-go packages or the JSON API.

true. but that's probably harder than just writing another module that does the right thing. ;)

  1. This is already possible using paclink-unix.

that thing again. ;) maybe i should just bite the bullet and try it out, but i'm not sure i want to go back to coding in C and learning yet another thing...

martinhpedersen commented 7 years ago

Any reference for that Python script? I'm fluent in python as well and could give it a shot.

I have sent an email to the author asking if he will share the script and/or join in on the discussion :)

I guess avoiding the JSON interface would have some benefits. The main reason for using it would be that it could be implemented in any language and still benefit from Pat's (de)serialization of the Winlink Message Structure (let's call it WMS from now on). The great advantage of implementing the tool in Go using the wl2k-go packages is of course that it would be more likely to be merged into Pat.

... maybe i should just bite the bullet and try it out, but i'm not sure i want to go back to coding in C and learning yet another thing...

I'd rather you practice you Go skills ;-)

So, let's discuss how we could implement this proposal as a Go tool (or Pat sub-command).

A simple pat-sendmail tool could possibly use the net/mail package to parse incoming emails, copy the content into a fbb.Message (here be dragons) and write it to Pat's mailbox using the DirHandler.AddOut method.

For the reverse (incoming WMS), we could monitor the mailbox-folder using github.com/fsnotify/fsnotify and read incoming messages from the inbox using the mailbox.OpenMessage function, construct a proper email message from the fbb.Message (here be dragons too) and deliver it to the local MTA message queue (using SMTP?).

Does this resemble what you had in mind?