Open Nerixyz opened 2 years ago
Where does stream.map(...)
come from?
44 | .streaming(stream.map(|b| Ok::<_, io::Error>(b.freeze())))
| ^^^ `AsyncPipeReader` is not an iterator
|
::: async_pipe.rs:9:1
|
9 | pub struct AsyncPipeReader(Arc<Mutex<Pipe>>);
| --------------------------------------------- doesn't satisfy `AsyncPipeReader: Iterator`
Where does
stream.map(...)
come from?
It's from futures::stream::StreamExt
.
maybe something for actix-web-lab
had a play with it and come up with something similar to the body::channel
ideas:
I think these might be two issues in one:
AsyncWrite
into aStream<Item=Bytes(Mut)>
(maybe something for actix-web-lab).Note: Also see discussion on actix-web Discord: https://discord.com/channels/771444961383153695/771447545154371646/1009110232473014383
The proposed example is one that streams files from a directory (
files
) as a zip file, i.e. it dynamically creates the zip file.For this I'm using
async_zip
which exposes an API that requires the user to pass in anAsyncWrite
(ZipFileWriter::new
). To "pipe" theAsyncWrite
to aStream
, I'm using aDuplexStream
and theBytesCodec
.main.rs
```rust use actix_web::{get, http, App, HttpResponse, HttpServer, Responder}; use async_zip::write::{EntryOptions, ZipFileWriter}; use futures::stream::TryStreamExt; use std::io; use tokio::io::AsyncWrite; use tokio_util::codec; #[get("/")] async fn index() -> impl Responder { let (to_write, to_read) = tokio::io::duplex(2048); tokio::spawn(async move { let mut zipper = async_zip::write::ZipFileWriter::new(to_write); if let Err(e) = read_dir(&mut zipper).await { // TODO: do something eprintln!("Failed to write files from directory to zip: {e}") } if let Err(e) = zipper.close().await { // TODO: do something eprintln!("Failed to close zipper: {e}") } }); let stream = codec::FramedRead::new(to_read, codec::BytesCodec::new()).map_ok(|b| b.freeze()); HttpResponse::Ok() .append_header(( http::header::CONTENT_DISPOSITION, r#"attachment; filename="folder.zip""#, )) // not sure if this is really necessary, // but we're already sending compressed data, // so make sure other middleware won't compress this again .append_header((http::header::CONTENT_ENCODING, "identity")) .streaming(stream) } async fn read_dirAs I explained on Discord, using a
DuplexStream
is probably overkill, since it's supposed to be used bi-directional (see example in tokio docs), so I tried to extract the internalPipe
used by the tokio implementation and made a pipe specifically for (buffered) piping ofAsyncWrite
to aStream<BytesMut>
. I'm not sure if this should be included inactix-web-lab
as a utility when dealing withAsyncWrite
(or maybe in some other crate?).async_pipe.rs
```rust use std::{sync::{Arc, Mutex, MutexGuard}, task::{self,Waker, Poll}, pin::Pin}; use bytes::{BytesMut, Buf}; use futures::Stream; use tokio::io::AsyncWrite; /// The `AsyncWrite` half of an [`async_pipe`] pub struct AsyncPipeWriter(Arcindex handler with async_pipe
```rust #[get("/")] async fn index() -> impl Responder { let (to_write, stream) = async_pipe(2048); tokio::spawn(async move { let mut zipper = async_zip::write::ZipFileWriter::new(to_write); if let Err(e) = read_dir(&mut zipper).await { // TODO: do something eprintln!("Failed to write files from directory to zip: {e}") } if let Err(e) = zipper.close().await { // TODO: do something eprintln!("Failed to close zipper: {e}") } }); HttpResponse::Ok() .append_header(( http::header::CONTENT_DISPOSITION, r#"attachment; filename="folder.zip""#, )) // not sure if this is really necessary, // but we're already sending compressed data, // so make sure other middleware won't compress this again .append_header((http::header::CONTENT_ENCODING, "identity")) .streaming(stream.map(|b| Ok::<_, io::Error>(b.freeze()))) } ```cargo.toml
```toml [package] name = "actix-zippy" version = "0.1.0" edition = "2021" [dependencies] actix-web = "4.1.0" async_zip = "0.0.8" bytes = "1.2.1" futures = "0.3.23" tokio = { version = "1.20.1", features = ["io-util", "fs"] } tokio-stream = "0.1.9" tokio-util = { version = "0.7.3", features = ["codec"] } ```