Encoding Str/Cool/etc. is a pain for middleware

[zostay]

While working on porting some of the middleware from Plack to Smack, I've noticed that the process of encoding stringified bits is a pain point, particularly for middleware. Obviously, most middleware won't have this problem as it won't deal with the body at all, but many of the Plack middleware apps handle the body to do things like chunked transfer encoding or counting bytes to add the Content-Length, etc.

Most other middleware is shorter and cleaner in Smack than Plack, but these ones that deal with the body are clearly more complex. It would be nice to come up with some idiom for avoiding this or address it in the standard somehow.

[muraiki]

I've tried searching for Smack but couldn't find any info on it... could you link a site to me? Thanks!

[zostay]

Sorry, I probably should have linked it in the issue description:

https://github.com/zostay/Smack

[zostay]

Some additional details and thoughts:

This is one of those places where I think Perl 6 is really doing the right thing in the way it handles encoding, allowing it to be explicit as soon as you need bytes. I initially considered requiring that apps return blobs, but that makes encoding explicit and I think encoding should be implicit unless you actually care about it.

An easy solution is to just provide a function that can be applied as a Supply.map() to automatically provide the encoding in Smack::Util or some such place.

use Smack::Util;
sub mw(%env) {
    callsame().then(-> $p {
        my ($s, $h, Supply(Any) $body) = $p.result;
        $body.=map(&Smack::Util::stringify-encode);
        ...; # do interesting middleware-y stuff
        $s, $h, $body
    });
}

For performance reasons, however, I don't really want to have this map layered in with each middleware that needs to work with the body.

[zostay]

I realized this evening that this would be easily solved if there was a built in Supply that enforced an of type constraint. I don't think there is though. I may try to code one and propose a PR on rakudo because it would make this doable without much fuss.

[zostay]

So, I think I may have a solution to this that elegantly mirrors the discussion in #14. In that issue, we're discussing the consequences of having the server provide the request body as a Supply. In that case, the server is responsible for providing a stream of Blobs, but middleware might be used to transform those Blobs into encoded strings, or custom message objects after processing incoming WebSocket frames, etc.

If we apply similar thinking here, we can require that the server be able to process the emitted objects as already specified: either as binary Blobs or as objects that must be stringified and encoded. However, the application is not required to emit that at all and may depend on middleware to map the response Supply into something that may be processed by the server.

Clearly, in both directions, the middleware must clearly state it's expectations in documentation if not in it's interface and generic middleware will have to operate closer to the server than the application in such cases, but that just means the order the middleware is applied is significant, which is not really a problem.

Making the request body/response body into a chain of Supply objects like this is appealing.

[zostay]

I may have made this more painful rather than less, but I've now modified the spec so that an application is permitted to emit any kind of object so long as some middleware is able to map the application emitted objects into something acceptable to the server.

I have also added List of Pairs and Associative objects to the list of things the server is expected to process. This is because handling trailing headers in HTTP/2 requires the server to perform special framing, which will break if we just try to include the headers as part of the regular payload. I have added the Associative to allow any kind of protocol-specific message to be sent to the server. I have not really fleshed out or explained what these are yet.