The http crate not having its own body type or trait is painful

[jimmycuadra]

Perhaps a better issue to create in https://github.com/hyperium/http, but I thought it might be worth discussing here first before proposing something over there.

http provides request and response types that take a generic parameter for the body of the request/response, but this parameter is not bounded by any traits, so it could be anything at all. As a result, every crate that consumes these types needs to create its own body type and be aware of other crates' body types in order to provide conversions if necessary. For example, tide's body type has explicit awareness of hyper's body type, and an impl for converting bodies between types.

If you want to make an HTTP utility that uses http's request and response types, you can't really make it agnostic to HTTP client library because of the body compatibility issue. We've run into this problem in https://github.com/ruma/ruma-api. Initially we just supported Vec<u8> as the single body type, since it's the most "generic" concrete type that would work, but eventually gave up and just used hyper's body type.

I don't recall the exact reason that the authors of http decided not to commit to either a concrete body type or a trait bound that would provide some interoperability between body types. I also vaguely recall reading something about a future version of the crate possibly needing to change this decision, but I can't find it now.

Hyper has a Payload trait which is its common denominator for body types, and of course a concrete body type of its own. But all this allows you to do is make a custom body type when working with hyper. Since Payload is in hyper and not http, it doesn't help with http-client-agnostic code. In Tide, the main purpose of the body type seems to be to wrap the two possibilities of streaming and non-streaming responses. Perhaps this is enough flexibility for crate authors that it could be baked into http proper, in which case the proliferation of body types with explicit conversion impls might be halted.

Other ideas?

[yoshuawuyts]

@jimmycuadra I've heard this come up before, and am sympathetic to what you're experiencing. Opening up an issue on hyperium/hyper might indeed be the best way to go for now. It'd be great if we could figure out a path forward for this.

[yoshuawuyts]

An update on this thread: the working group has created the http-service crate specifically to deal with this problem. It provides (streaming) request and response types. I hope it comes in useful!

[jimmycuadra]

Awesome! Thanks so much, Yoshua!

[jimmycuadra]

I'm gonna reopen this because the authors of the http crate don't want to use http-service (see https://github.com/hyperium/http/issues/298) and so I'm back to square one on the original problem. Any ideas for how to get http-service adopted by other libraries?

[zmlzm]

I always think the HTTP libray should be in rustasync organization

[LucioFranco]

@jimmycuadra one reason there is no body type in the http crate is due to the fact that the http crate isnt async or sync, while a body type needs to pick one. I think sean has done a lot of work in figuring out what body type works well in the use case of hyper and we have created a https://github.com/hyperium/http-body crate that contains a trait around what an async http body should implement. This trait is already currently implemented for hyper::Body. I think this should solve your issues?

[jimmycuadra]

Thank you, @LucioFranco, I was unaware of http-body. It sounds like a well considered design for async code, but as you mention, the http crate not committing to a type or bound is to keep from locking into this choice as well. I might be remembering incorrectly, but tide/http-service used to contain a body type that supported both async and sync code, but seems to have eschewed this in favor of async-exclusive types. http-service even has a sub-crate specifically for hyper support, which includes conversions from hyper's body type to its own, which is exactly what I'm hoping to avoid. In any case, the fact that hyper and tide still use different foundations for their request and response bodies illustrates that the problem still exists.

I think this is largely a social problem and not a technical one (which is why it makes sense to discuss in the async working group as opposed to on the issue tracker of a specific library). If there were some agreement among HTTP libraries on which of these abstractions to share, then it would be easier for my project to choose a more interoperable foundation. As it stands, it seems that we either have to pick one of these foundational libraries that has not seen any adoption beyond its own project (see https://crates.io/crates/http-body/reverse_dependencies and https://crates.io/crates/http-service/reverse_dependencies), or use something extremely generic for our body type like Vec<u8> which would forego a lot of the benefit the traits/types from these other projects would bring, and would also forego some async use cases.

CC @carllerche @seanmonstar, would appreciate your thoughts if you have time.

[LucioFranco]

@jimmycuadra and actually to continue it looks like http-service's body is very very similar to https://docs.rs/tokio-buf/0.1.1/tokio_buf/trait.BufStream.html which is something that is also trying to achieve the same goal. We have been trying to provide types that can be used outside of hyper,tokio, tower so that the ecosystem as a wide can benefit. I totally feel your pain about the body types and its a big reason why I pushed hard for http-body to be released.

I think another big issue is that a lot of tide an http-service are based around futures 0.3 while we are just now moving now. So http-body is still in its futures 0.1 phase which makes it hard to work with in http-service and that may be a huge reason why there are two body types and there is compating going on. That said, I am currently in the process of updating http-body and would love to see how we could get some of that stuff working in http-service.