Generic streaming interface

[endgame]

I wonder if we can break the direct dependency on conduit. paginateEither really only needs some kind of function to yield into.

On the consumption side, we really just need an action m (Maybe ByteString) which we can repeatedly invoke, and can build that out of unconsM or your streaming library's equivalent.

[brendanhay]

There was originally some movement around removing it - but Route53's (and S3 at one point) XML interface use{d.s} xml-conduit so the transitive dependency existed anyway. If that is rendered unnecessary the next step is generate data types that are free over some body type variable (or m body, whatever). That is, requests need it in addition to responses:

data PutObject body = PutObject'
  { acl :: Prelude.Maybe ObjectCannedACL,
    bucketKeyEnabled :: Prelude.Maybe Prelude.Bool,
    cacheControl :: Prelude.Maybe Prelude.Text,
    contentDisposition :: Prelude.Maybe Prelude.Text,
    contentEncoding :: Prelude.Maybe Prelude.Text,
...
    body :: body
  }

[endgame]

Neither amazonka-route53 nor amazonka-s3 seem to depend on xml-conduit these days, but a more fundamental direct dependency is amazonka-core's reliance on http-conduit, which I can't see us easily fixing. And since most of the HTTP stuff is built on Snoymanverse libraries I don't see us being able to provide a completely streaming-agnostic interface here. The best thing we can do, I think, is write wrappers in separate libraries (amazonka-streaming, amazonka-pipes, except that latter name is used for EventBridge Pipes) which convert to conduits as the final step before handing the streaming body off to amazonka.