Support for large emails via smart disk buffering?

[xeoncross]

I want to support accepting large SMTP DATA bodies without having to store the body in memory. Using a disk buffer or "memory-constrained buffer" would allow the server to accept larger bodies from multiple clients concurrently.

https://github.com/mhale/smtpd/blob/master/smtpd.go#L603

// Buffer up to ~6MB in RAM and limit max buffer size to ~200MB
multibuf.New(r, multibuf.MemBytes(1024 * 1024 * 6), multibuf.MaxBytes(1024 * 1024 * 200))

[mhale]

Great idea. I'll have a go at implementing it over the holiday period.

[xeoncross]

Actually, maybe instead of disk buffering: consider just passing the connection io.reader to a mime parser that can handle streaming decoding:

• https://github.com/emersion/go-message
• ( or my almost finished) https://github.com/Xeoncross/mimestream

This could be optional by having two handlers - the []byte one you have now, or a second optional handler that took an io.Reader.

type MailHandler interface {
    Handle(origin net.Addr, from string, to []string, data []byte)
    HandleStreaming(origin net.Addr, from string, to []string, data io.Reader)
}

...

type MyHandler struct {}

func (m MyHandler) Handle(origin net.Addr, from string, to []string, data []byte) {}
// or 
func (m MyHandler) HandleStreaming(origin net.Addr, from string, to []string, data io.Reader) {}

func main() {
    smtpd.ListenAndServe("127.0.0.1:2525", MyHandler{}, "MyServerApp", "")
}

[mhale]

Passing io.Reader to the handler would still require accepting all the data into memory, because it's readData() that would need to be replaced with a streaming MIME parser in this scenario.

Perhaps a custom DATA handler is required, in the same way that there is a custom RCPT handler?

[xeoncross]

Yes, that was what I meant. Replace the handler function with a struct like above, then replace lines 357-387 with an if check to do what is need.

Perhaps a custom reader that wrapped the connection and wrote what it read to the streaming mime parser. (not working, just a quick example)

Type DataStreamer struct {
    w io.Writer
    s *session
    total int
}

func (w *DataStreamer) Read(p []byte) (n int, err error) {

    data := make([]byte, len(p))

    for {
        if w.s.srv.Timeout > 0 {
            w.s.conn.SetReadDeadline(time.Now().Add(w.s.srv.Timeout))
        }

        n, err = w.s.br.Read(data)
        if err != nil {
            return 0, err
        }

        // Handle end of data denoted by lone period (\r\n.\r\n)
        if bytes.Equal(data, []byte(".\r\n")) {
            break
        }

        // Remove leading period (RFC 5321 section 4.5.2)
        if string(data[0]) == '.' {
            data = data[1:]
        }

        // Enforce the maximum message size limit.
        if w.s.srv.MaxSize > 0 {
            if len(data)+w.total > w.s.srv.MaxSize {
                _, _ = w.s.br.Discard(s.br.Buffered()) // Discard the buffer remnants.
                return o, maxSizeExceeded(w.s.srv.MaxSize)
            }
        }

        // Actually write to the writer below
        w.w.Write(p)
    }
    return
}

ds := &DataStramer{s: s, w: mimeparserhere}

Edit: What in the world did I just write. You would either implement an io.Reader/io.Writer or use a loop - not both.

[xeoncross]

I had time today and went ahead and added streaming multipart/mime body support as a test everything seems to be working. I still have more to do but wanted to update here in case you want to borrow any ideas.

Unfortunately, this can't be integrated back as a PR because I removed all the authentication code/tests which made up about half of the codebase. I am looking to run a public server and have no use for AUTH requests.

[mhale]

I am interested in this streaming attachment idea. Thanks for following up.

Btw, AUTH commands return 502 not implemented if no AuthHandler is defined. They'd be rejected in the default configuration.

[dubcanada]

@Xeoncross - I am interested in this as well I noticed @mhale has not done this yet.

Do you have intentions of maintaining the fork or should we work towards integrating your work with this repo?

[xeoncross]

@dubcanada I'm not currently using my fork, so I don't have any plans to maintain it. Work on integrating it into this project would be great. That said, it's also mostly complete as-is in my fork, so it should be immediately usable.

I would checkout https://github.com/emersion?tab=repositories repositiories as well. I've forked, tested and compared dozens of Go libraries in the "email" space and there is no one that compares to the work he's done. He's forked the official SMTP library and written DKIM, MIME, WebDAV, SMTP and many other libraries. So check his work out for more information.

Even with my fork of this project adding streaming and proper use of textproto I was still unable to beat his more feature-complete version.

[mhale]

Closing due to age.