Open sfackler opened 2 years ago
eholk
commented
2 years ago We discussed this in the wg-async triage today. We think the best course of action is to try to get a minimal test case for this. Afterwards, we'll probably want to get T-Types input on this. @vincenzopalazzo is going to try to reproduce this and minimize the test case.
eholk
commented
2 years ago @rustbot label AsyncAwait-Triaged
vincenzopalazzo
commented
2 years ago @rustbot claim
danielhenrymantilla
commented
2 years ago I wonder if the following snippet wouldn't be a reduction:
struct Type<'a, 'b>(&'a &'b (), ::core::marker::PhantomData<*mut ()>);
// Note: if this had an *implicit* `'b : 'a` things would work.
unsafe impl<'b : 'a, 'a> Send for Type<'a, 'b> {}
fn foo() -> impl Send { async {
let local = ();
let r = &&local;
async {}.await;
let it = Type(r, <_>::default());
async {}.await;
}}@sfackler - Would it be possible to share more of the surrounding code for this example so we can see more of what's going on?
sfackler
commented
2 years ago I'm working on minimizing it.
sfackler
commented
2 years ago Here is a semi-minimized repro. It still depends on Hyper, but it could probably be replaced with some PhantomData inserts with some more work: https://gist.github.com/sfackler/a7f3aaf2ce3f5ff4c99e1a3d67f828ed
error: higher-ranked lifetime error
--> witchcraft-server/src/lib.rs:162:5
|
162 | / spawn(async move {
163 | | let _ = handle_service.call(stream).await;
164 | | });
| |______^
|
= note: could not prove `for<'r> impl for<'r> futures_util::Future<Output = ()>: Send`For whatever reason, the presence of the AuditLogService in the future seems to be a trigger. If you change line 160 to just let request_service = HandlerService it compiles.
vincenzopalazzo
commented
2 years ago Thank @sfackler,
I think the core concept missed in your previous report are:
Using a wrapper around Future like
pub struct AdaptorFuture<F> {
inner: F,
}
impl<F, T> Future for AdaptorFuture<F>
where
F: Future<Output = T>,
{
type Output = Result<T, io::Error>;
fn poll(self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll<Self::Output> {
panic!()
}
}and your handle_service implementation
impl<S, R> hyper::service::Service<R> for AdaptorService<S>
where
S: Service<R>,
{
type Response = S::Response;
type Error = io::Error;
type Future = AdaptorFuture<S::Future>;
fn poll_ready(&mut self, _: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, req: R) -> Self::Future {
AdaptorFuture {
inner: self.inner.call(req),
}
}
}@sfackler I'm trying to reproduce your error without hyper, and till now I had no luck.
Maybe you can help with some tips? I had the code there https://github.com/vincenzopalazzo/rio/pull/13
sfackler
commented
2 years ago Not sure I have a better idea of how to minimize than you, unfortunately.
brandonros
commented
2 years ago I think I was able to recreate it?
It took me like 35 minutes to figure out where it was because the compiler only gives this message with nothing else and no extra details:
error: higher-ranked lifetime error
--> bot/src/main.rs:141:23
|
141 | let handle2 = tokio::task::spawn(robinhood_task());
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: could not prove `impl Future<Output = ()>: Send`
This looks promising @brandonros! I will give it a shot and also try to reproduce inside https://github.com/vincenzopalazzo/rio/pull/13
Looks like that when the runtiime try to span the following task there is an error like the one reported from you
async fn robinhood_task() {
let robinhood = Robinhood::new();
let token = String::from("fake");
let instrument_ids = vec![String::from("fake")];
robinhood.get_options_market_data(token, instrument_ids).await;
}In particulat I play a little bit with it and I found the following example to reproduce the error
use futures::StreamExt;
use log::info;
use serde::{Deserialize, Serialize};
pub struct Robinhood;
impl Robinhood {
pub fn new() -> Robinhood {
return Robinhood {};
}
async fn foo(&self, t: String, b: Vec<String>) -> Vec<String> {
// Looks like that the problem is generated from the following code
let b: Vec<&[String]> = b.chunks(2).collect();
let futures = b.into_iter().map(|it| {
return async { vec![] };
});
let results: Vec<String> = futures::stream::iter(futures)
.buffer_unordered(4)
.collect::<Vec<_>>()
.await
.into_iter()
.flatten()
.collect();
results
}
}
async fn foo_ok(t: String, b: Vec<String>) -> Vec<String> {
let b: Vec<&[String]> = b.chunks(2).collect();
let futures = b.into_iter().map(|it| {
return async { vec![] };
});
let results: Vec<String> = futures::stream::iter(futures)
.buffer_unordered(4)
.collect::<Vec<_>>()
.await
.into_iter()
.flatten()
.collect();
results
}
async fn robinhood_task() {
let robinhood = Robinhood::new();
foo_ok(String::new(), vec![]).await;
robinhood.foo(String::new(), vec![]).await;
}
fn main() {
async {
let handle1 = tokio::task::spawn(robinhood_task());
let _ = tokio::join!(handle1);
};
}@rustbot label +A-async-await
our penguing triage bot it is not smart enough to reassign this in the next triage meeting so I need to do this trick!
rustbot
commented
2 years ago brandonros
commented
2 years ago I'm sure this is obvious to you, I just want to make sure it's documented somewhere:
switching let b: Vec<&[String]> = b.chunks(2).collect(); to let b: Vec<Vec<String>> = b.chunks(2).map(|c| c.to_vec()).collect(); works around the issue.
awesomelemonade
commented
1 year ago Unsure if this is the same exact issue, but I ran into a "higher-ranked lifetime error" when running https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=e8d2668ceca75a8ebf4f0c16cee04958. I tried to work around the failure by boxing it like suggested, but it still doesn't seem to work. The full error is
error: higher-ranked lifetime error
--> src/lib.rs:25:18
|
25 | let handle = tokio::spawn(accept_connection(rpc));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: could not prove `impl futures::Future<Output = Result<(), anyhow::Error>>: std::marker::SendEDIT: if I pin & box the lol future, it works: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=5d7d8b1dd6542571425b37b5e2cd6ee9
Tails
commented
1 year ago Had the same problem, but @awesomelemonade's example helped. It worked by pinning the try_join_all call and explicitly specifying the type in the assignment. Note that the Send bound made the difference in 'dyn Future<> + Send'.
// let objects = futures::future::try_join_all(futs)
// .await?
// .into_iter()
// .collect::<HashMap<Self::Address, C>>();
let f: Pin<
Box<
dyn Future<Output = anyhow::Result<Vec<(<Self as StorageProvider>::Address, C)>>>
+ Send,
>,
> = Box::pin(futures::future::try_join_all(futs));
f.await;@Tails @awesomelemonade you can skip the heap allocation with a helper function:
fn assert_send<'u, R>(fut: impl 'u + Send + Future<Output = R>)
-> impl 'u + Send + Future<Output = R>
{
fut
}
let f = assert_send(futures::future::try_join_all(futs));
f.awaitAll this, and more is present in that Discord discussion I mentioned:
mcronce
commented
1 year ago I was running into this problem in a project tonight (which prompted me to go through a bunch of its dependencies and enable warn(clippy::future_not_send) :joy:); the assert_send() (and, indeed, Box::pin) solution didn't work, but switching to stable did. Not sure whether or not it's well known that this is a nightly-specific problem, or if I'm just hitting it in a nightly-specific way, so take that for what it's worth.
I did update my nightly to latest, but I'm not sure what version I was on before. Not something more than a few days old, though, I'm pretty sure.
kylezs
commented
1 year ago In my case, code was compiling in nightly-2022-08-08 but when bumping to nightly-2022-12-16 it started failing with this error.
The future was carrying a stream (that was Send) across an await boundary. So inspired by some solutions above, I had to Box::pin the stream (which was already the case before I bumped the Rust version) and use the assert_stream_send below. So it looked like assert_stream_send(Box::pin(stream))
fn assert_stream_send<'u, R>(
strm: impl 'u + Send + Stream<Item = R>,
) -> impl 'u + Send + Stream<Item = R> {
strm
}Then it compiled.
hseeberger
commented
1 year ago Interesting. In my case I also have a future producing a stream, but I cannot get it to compile:
#![allow(incomplete_features)]
#![feature(return_position_impl_trait_in_trait)]
use anyhow::Result;
use futures::{pin_mut, Future, Stream, StreamExt};
use tokio::task;
#[tokio::main]
async fn main() -> Result<()> {
Ok(())
}
async fn run<T>(mut foo: T) -> Result<()>
where
T: Foo + Send + 'static,
{
task::spawn(async move {
let bar = assert_send(foo.bar());
let bar = bar.await; // `impl Stream<Item = i32> + Send`
let mut bar = stream_assert_send(Box::pin(bar)); // still `impl Stream<Item = i32> + Send`
// higher-ranked lifetime error
// bar.next().await;
let next = assert_send(bar.next());
// Also higher-ranked lifetime error
next.await;
});
Ok(())
}
fn assert_send<'a, T>(
fut: impl std::future::Future<Output = T> + Send + 'a,
) -> impl std::future::Future<Output = T> + Send + 'a {
fut
}
fn stream_assert_send<'a, T>(
stream: impl futures::Stream<Item = T> + Send + 'a,
) -> impl futures::Stream<Item = T> + Send + 'a {
stream
}
trait Foo {
fn bar<'a>(&'a mut self) -> impl Future<Output = impl Stream<Item = i32> + Send + 'a> + Send;
}
frxstrem
commented
1 year ago I'm having a similar issue, I've boiled it down to this minimal reproducible example:
use futures::{future::BoxFuture, stream, StreamExt};
async fn inner() {
stream::iter(&Vec::<()>::new())
.map(|_| stream::empty::<()>())
.flatten()
.collect::<()>()
.await;
}
fn outer() -> BoxFuture<'static, ()> {
// error: higher-ranked lifetime error
Box::pin(async move { inner().await })
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
// = note: could not prove `Pin<Box<impl futures::Future<Output = ()>>>: CoerceUnsized<Pin<Box<
// (dyn futures::Future<Output = ()> + std::marker::Send + 'a)>>>`
}The issue seems to be related to the closure |_| stream::empty::<()>(), since replacing the closure with a function pointer works:
fn f(_: &()) -> stream::Empty::<()> {
stream::empty()
}
stream::iter(&Vec::<()>::new())
.map(f)
.flatten()
.collect::<Vec<_>>()
.await;Additionally, using the assert_send_stream technique as mentioned above seems to work as well:
assert_stream_send(
stream::iter(&Vec::<()>::new())
.map(|_| stream::empty::<()>())
)
.flatten()
.collect::<Vec<_>>()
.await;Versions and dependencies:
rustc 1.66.0 (69f9c33d7 2022-12-12)
futures = "0.3.25"
emersonford
commented
1 year ago Here's another minimal reproducible example, with no dependencies:
use std::future::Future;
async fn foo() {
let some_vec = Vec::<()>::new();
let flat = some_vec.iter()
.map(|_| std::iter::empty::<()>())
.flatten();
std::mem::drop(flat);
async{}.await;
}
fn assert_send<'u, R>(fut: impl 'u + Send + Future<Output = R>)
-> impl 'u + Send + Future<Output = R>
{
fut
}
fn main() {
let fut = assert_send(foo());
}desugared (think it's a bit clearer?):
use std::future::Future;
fn foo() -> impl Future<Output = ()> + Send {
let some_vec = Vec::<()>::new();
async move {
let flat = some_vec.iter()
.map(|_| std::iter::empty::<()>())
.flatten();
std::mem::drop(flat);
async{}.await;
}
}what's critical here is the flatten(). It looks like something has to take ownership of the Iterator from .map to trigger the error?
EDIT: for clarification, these will spit out the following error:
error: implementation of `Iterator` is not general enough
--> src/main.rs:8:5
|
8 | / async move {
9 | | let flat = some_vec.iter()
10 | | .map(|_| std::iter::empty::<()>())
11 | | .flatten();
12 | | std::mem::drop(flat);
13 | | dummy().await;
14 | | }
| |_____^ implementation of `Iterator` is not general enough
|
= note: `Iterator` would have to be implemented for the type `std::slice::Iter<'0, ()>`, for any lifetime `'0`...
= note: ...but `Iterator` is actually implemented for the type `std::slice::Iter<'1, ()>`, for some specific lifetime `'1`
error: implementation of `FnOnce` is not general enough
--> src/main.rs:8:5
|
8 | / async move {
9 | | let flat = some_vec.iter()
10 | | .map(|_| std::iter::empty::<()>())
11 | | .flatten();
12 | | std::mem::drop(flat);
13 | | dummy().await;
14 | | }
| |_____^ implementation of `FnOnce` is not general enough
|
= note: closure with signature `fn(&'0 ()) -> std::iter::Empty<()>` must implement `FnOnce<(&(),)>`, for any lifetime `'0`...
= note: ...but it actually implements `FnOnce<(&(),)>`which is identical to what's found in #71671. This originally doesn't appear related to the higher-ranked lifetime error error, but I'm fairly certain this error is the root cause for this because you can desugar what @frxstrem came up with like so:
use std::future::Future;
use futures::{future::BoxFuture, stream, StreamExt};
async fn dummy() {}
fn inner() -> impl Future<Output = ()> {
let some_vec = Vec::<()>::new();
async move {
let flat = stream::iter(&some_vec)
.map(|_| stream::empty::<()>())
.flatten();
std::mem::drop(flat);
dummy().await;
}
}
fn outer() -> BoxFuture<'static, ()> {
Box::pin(async move { inner().await })
}to get this error involving the iterator.
EDIT2: this only happens when you need a Send bound on the future, explaining why folks are running into this with tokio::spawn (BoxFuture from the futures crate also requires a Send bound). If you don't need a Send bound, then you don't get this error.
EDIT3: looks like it's related to #64552 as well?
EDIT4: based on https://github.com/dtolnay/async-trait/issues/215, this is definitely just the same issue as #64552.
danielhenrymantilla
commented
1 year ago So, from @emersonford's snippet:
let _: &dyn Send = &async {
let _it =
[()].iter()
.map(|it| None::<()>)
.flatten()
;
async {}.await;
};I have tried to figure out what was about .flatten() which would cause this, _when it doesn't happen for .flat_map()!_
.flatten() and .flat_map() delegate to the same inner type)It turns out, Flatten is defined in terms of an associated type (the Iterator::Item, which when dealing with .map(), involves a FnOnce::Output), whereas FlatMap introduces a new free type parameter which is then constrained to be equal to the associated type later on.
From that observation, I was able to craft an even smaller reduction, since it involves no Iterator shenanigans:
trait Trait { type Assoc; }
struct Foo<T : Trait>(T::Assoc);
impl Trait for fn(&'static ()) {
type Assoc = ();
}
let _: &dyn Send = &async {
let _it = Foo::<fn(&'static ())>(());
async {}.await;
};Note that the argument of fn() involving a lifetime, even if it's a hard-coded lifetimes such as 'static, is key to trigger this.
emersonford
commented
1 year ago Just wanted to share some things I noticed debugging this yesterday, using this code block:
use std::future::Future;
fn foo() -> impl Future<Output = ()> + Send {
let some_vec = Vec::<()>::new();
async move {
let flat = some_vec.iter()
.map(|_| std::iter::empty::<()>())
.flatten();
std::mem::drop(flat);
async{}.await;
}
}I was curious why dropping + Send on impl Future makes this compile, so I spit out debug logs from the compiler for both a + Send version and no + Send version to figure out where they differ. As far as I can tell, this is the where they begin to differ:
Non + Send version:
│ │ ├─┐rustc_borrowck::type_check::canonical::normalize_with_category value=([[async block@lib.rs:6:5: 12:6]]; c_variadic: false)->[async block@lib.rs:6:5: 12:6], location=bb1[3], category=Boring
│ │ │ ├─0ms DEBUG rustc_borrowck::type_check::canonical output=([[async block@lib.rs:6:5: 12:6]]; c_variadic: false)->[async block@lib.rs:6:5: 12:6], constraints=None
│ │ ├─┘
│ │ ├─┐rustc_borrowck::type_check::canonical::normalize_with_category value=impl std::future::Future<Output = ()>, location=bb1[3], category=Boring
│ │ │ ├─0ms DEBUG rustc_borrowck::type_check::canonical output=impl std::future::Future<Output = ()>, constraints=None
│ │ ├─┘
│ │ ├─┐rustc_borrowck::type_check::relate_tys::relate_types a=impl std::future::Future<Output = ()>, v=-, b=[async block@lib.rs:6:5: 12:6], locations=Single(bb1[3]), category=Return(Normal)
│ │ │ ├─0ms DEBUG rustc_borrowck::type_check::canonical output=(), constraints=None
│ │ ├─┘Send version:
│ │ ├─┐rustc_borrowck::type_check::canonical::normalize_with_category value=([[async block@lib.rs:6:5: 12:6]]; c_variadic: false)->[async block@lib.rs:6:5: 12:6], location=bb1[3], category=Boring
│ │ │ ├─0ms DEBUG rustc_borrowck::type_check::canonical output=([[async block@lib.rs:6:5: 12:6]]; c_variadic: false)->[async block@lib.rs:6:5: 12:6], constraints=None
│ │ ├─┘
│ │ ├─┐rustc_borrowck::type_check::canonical::normalize_with_category value=impl std::future::Future<Output = ()> + std::marker::Send, location=bb1[3], category=Boring
│ │ │ ├─0ms DEBUG rustc_borrowck::type_check::canonical output=impl std::future::Future<Output = ()> + std::marker::Send, constraints=None
│ │ ├─┘
│ │ ├─┐rustc_borrowck::type_check::relate_tys::relate_types a=impl std::future::Future<Output = ()> + std::marker::Send, v=-, b=[async block@lib.rs:6:5: 12:6], locations=Single(bb1[3]), category=Return(Normal)
│ │ │ ├─0ms DEBUG rustc_borrowck::type_check::canonical output=(), constraints=Some(QueryRegionConstraints { outlives: [(Binder(OutlivesPredicate('_#2r, RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) })), []), BoringNoLocation), (Binder(OutlivesPredicate('_#3r, RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) })), []), BoringNoLocation), (Binder(OutlivesPredicate('_#4r, RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) })), []), BoringNoLocation), (Binder(OutlivesPredicate('_#2r, RePlaceholder(Placeholder { universe: U1, name: BrAnon(1, Some(lib.rs:8:14: 8:17 (#0))) })), []), BoringNoLocation), (Binder(OutlivesPredicate(RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }), '_#2r), []), BoringNoLocation), (Binder(OutlivesPredicate(RePlaceholder(Placeholder { universe: U1, name: BrAnon(1, Some(lib.rs:8:14: 8:17 (#0))) }), '_#2r), []), BoringNoLocation), (Binder(OutlivesPredicate(RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }), '_#3r), []), BoringNoLocation), (Binder(OutlivesPredicate(RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }), '_#4r), []), BoringNoLocation)], member_constraints: [] })
│ │ │ ├─┐rustc_borrowck::type_check::push_region_constraints locations=Single(bb1[3]), category=Return(Normal)
│ │ │ │ ├─0ms DEBUG rustc_borrowck::type_check constraints generated: QueryRegionConstraints {
│ │ │ │ │ outlives: [
│ │ │ │ │ (
│ │ │ │ │ Binder(
│ │ │ │ │ OutlivesPredicate(
│ │ │ │ │ '_#2r,
│ │ │ │ │ RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }),
│ │ │ │ │ ),
│ │ │ │ │ [],
│ │ │ │ │ ),
│ │ │ │ │ BoringNoLocation,
│ │ │ │ │ ),
│ │ │ │ │ (
│ │ │ │ │ Binder(
│ │ │ │ │ OutlivesPredicate(
│ │ │ │ │ '_#3r,
│ │ │ │ │ RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }),
│ │ │ │ │ ),
│ │ │ │ │ [],
│ │ │ │ │ ),
│ │ │ │ │ BoringNoLocation,
│ │ │ │ │ ),
│ │ │ │ │ (
│ │ │ │ │ Binder(
│ │ │ │ │ OutlivesPredicate(
│ │ │ │ │ '_#4r,
│ │ │ │ │ RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }),
│ │ │ │ │ ),
│ │ │ │ │ [],
│ │ │ │ │ ),
│ │ │ │ │ BoringNoLocation,
│ │ │ │ │ ),
│ │ │ │ │ (
│ │ │ │ │ Binder(
│ │ │ │ │ OutlivesPredicate(
│ │ │ │ │ '_#2r,
│ │ │ │ │ RePlaceholder(Placeholder { universe: U1, name: BrAnon(1, Some(lib.rs:8:14: 8:17 (#0))) }),
│ │ │ │ │ ),
│ │ │ │ │ [],
│ │ │ │ │ ),
│ │ │ │ │ BoringNoLocation,
│ │ │ │ │ ),
│ │ │ │ │ (
│ │ │ │ │ Binder(
│ │ │ │ │ OutlivesPredicate(
│ │ │ │ │ RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }),
│ │ │ │ │ '_#2r,
│ │ │ │ │ ),
│ │ │ │ │ [],
│ │ │ │ │ ),
│ │ │ │ │ BoringNoLocation,
│ │ │ │ │ ),
│ │ │ │ │ (
│ │ │ │ │ Binder(
│ │ │ │ │ OutlivesPredicate(
│ │ │ │ │ RePlaceholder(Placeholder { universe: U1, name: BrAnon(1, Some(lib.rs:8:14: 8:17 (#0))) }),
│ │ │ │ │ '_#2r,
│ │ │ │ │ ),
│ │ │ │ │ [],
│ │ │ │ │ ),
│ │ │ │ │ BoringNoLocation,
│ │ │ │ │ ),
│ │ │ │ │ (
│ │ │ │ │ Binder(
│ │ │ │ │ OutlivesPredicate(
│ │ │ │ │ RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }),
│ │ │ │ │ '_#3r,
│ │ │ │ │ ),
│ │ │ │ │ [],
│ │ │ │ │ ),
│ │ │ │ │ BoringNoLocation,
│ │ │ │ │ ),
│ │ │ │ │ (
│ │ │ │ │ Binder(
│ │ │ │ │ OutlivesPredicate(
│ │ │ │ │ RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }),
│ │ │ │ │ '_#4r,
│ │ │ │ │ ),
│ │ │ │ │ [],
│ │ │ │ │ ),
│ │ │ │ │ BoringNoLocation,
│ │ │ │ │ ),
│ │ │ │ │ ],
│ │ │ │ │ member_constraints: [],
│ │ │ │ │ }
│ │ │ │ ├─┐rustc_borrowck::type_check::constraint_conversion::convert_all query_constraints=QueryRegionConstraints { outlives: [(Binder(OutlivesPredicate('_#2r, RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) })), []), BoringNoLocation), (Binder(OutlivesPredicate('_#3r, RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) })), []), BoringNoLocation), (Binder(OutlivesPredicate('_#4r, RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) })), []), BoringNoLocation), (Binder(OutlivesPredicate('_#2r, RePlaceholder(Placeholder { universe: U1, name: BrAnon(1, Some(lib.rs:8:14: 8:17 (#0))) })), []), BoringNoLocation), (Binder(OutlivesPredicate(RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }), '_#2r), []), BoringNoLocation), (Binder(OutlivesPredicate(RePlaceholder(Placeholder { universe: U1, name: BrAnon(1, Some(lib.rs:8:14: 8:17 (#0))) }), '_#2r), []), BoringNoLocation), (Binder(OutlivesPredicate(RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }), '_#3r), []), BoringNoLocation), (Binder(OutlivesPredicate(RePlaceholder(Placeholder { universe: U1, name: BrAnon(0, None) }), '_#4r), []), BoringNoLocation)], member_constraints: [] }
...
├─┐rustc_borrowck::region_infer::best_blame_constraint from_region='_#5r, from_region_origin=Placeholder(Placeholder { universe: U1, name: BrAnon(0, None) })I'm not at all familiar with what the compiler is doing so apologies if this is not at all correct but it seems like Send is triggering a constraint propagation down to the async block? And what for whatever reason, that round of constraint propagation isn't properly handling lifetimes...?
Oh also, you can trigger this same thing with + Sync instead of + Send, which makes me further think it's related to the constraint propagation code
Dessix
commented
1 year ago I'm currently working with axum and axum-server and have hit this issue in a multitude of ways on both nightly-2022-11-18 and nightly-2023-01-18, all of which when building or otherwise utilizing generic implementations of tower_service::Service.
Directly boxing the future resulting from axum_server::bind(...).serve(my_service) works, e.g.:
// Must reside in a non-async context, e.g. a function returning [BoxFuture]
axum_server::bind(addr).serve(service).map_err(From::from).boxed()But using an async block around it produces a block which fails to prove Send:
async move {
axum_server::bind(addr).serve(service).map_err(From::from).await
}.boxed()What's odd, and may give a clue as to what's going on, is that proving also fails if the surrounding block is async, even when directly boxing:
async move {
axum_server::bind(addr).serve(service).map_err(From::from).boxed().await
}.boxed() // This box fails to prove `Send` despite the prior succeedingEven more unusual, I noted some success when introducing BoxCloneService, but I've yet to discover a pattern with this, and it does not appear to consistently result in a fix on its own, nor produce further breakage. While I've yet to discern a pattern in what precise boxing addresses it, the most useful conclusion I've been able to draw is that async Send constraints appear to propagate further than their last .boxed() call and into the next synchronous context.
In summary, a workaround for axum users experiencing this issue (which may generalize further) is to box within a synchronous function returning BoxFuture, to (presumably) terminate the constraint propagation before it breaks.
emersonford
commented
1 year ago Just for reference, looks like this is the same issue as #71723.
EDIT: and #102870, #99492, #98380, #89976
(possibly #90656, #97515, #92415?)
vincenzopalazzo
commented
1 year ago Ok, I think it is time to bring it up again, and discuss it with the async-wg in the next triage meeting, there was good progress on how to reproduce it
@rustbot label +I-async-nominated
Swatinem
commented
1 year ago It is also a blocker to make progress on https://github.com/rust-lang/rust/pull/107562
msdrigg
commented
1 year ago Not sure if it is the same issue, but I am getting a similar error on stable 1.67.0
Error Here
error: higher-ranked lifetime error
--> crates/utils/src/supervisor_error.rs:13:9
|
13 | Box::pin(self.0.into_future())
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: could not prove `Pin<Box<impl futures::Future<Output = Result<(), Box<(dyn StdError + std::marker::Send + Sync + 'static)>>>>>: CoerceUnsized<Pin<Box<(dyn futures::Future<Output = Result<(), Box<(dyn StdError + std::marker::Send + Sync + 'b)>>> + std::marker::Send + 'c)>>>`Reproducible example here
use futures::{future::BoxFuture, Future, FutureExt};
pub async fn identity<F: Future>(future: F) -> F::Output {
future.await
}
type BoxError = Box<dyn std::error::Error + Send + Sync + 'static>;
pub struct Supervisor2(Supervisor1);
impl Supervisor2 {
fn into_future(&mut self) -> BoxFuture<'_, Result<(), BoxError>> {
Box::pin(self.0.into_future())
}
}
impl Supervisor1 {
pub async fn into_future(&mut self) -> Result<(), BoxError> {
let child = self.child.into_future().map(|res| res);
identity(child).await;
return Ok(());
}
}
pub struct Supervisor1 {
pub child: Box<Supervisor2>,
}There are several things changes that will keep the error from happening
BoxError type to something simpler like Box<String> stops the error from happening.
- type BoxError = Box<dyn std::io::Write + Send + Sync + 'static>;
+ type BoxError = Box<String>;.map(|res| res) also stops the error from happening.
- let child = self.child.into_future().map(|res| res);
+ let child = self.child.into_future();identity call stops the error as well
- identity(child).await;
+ child.await;I tried reducing it further, but I can't make any progress at the moment.
zeenix
commented
1 year ago FWIW, I hit this in zbus while trying to reduce the bloat from generics. :( Unfortunately the futures are quite complex in my case to be easily able to provide a reduced reproducer. :(
rustc 1.67.1 here as well.
rbtcollins
commented
1 year ago FWIW, I also hit this using streams, solved it with an explicitly typed Pin temporary variable around the entire combinator driven expression.
Munksgaard
commented
1 year ago I don't know if it is related, but I had a similar error using Rocket on rustc 1.70.0-nightly (f63ccaf25 2023-03-06):
error: higher-ranked lifetime error
--> src/main.rs:186:1
|
186 | #[get("/")]
| ^^^^^^^^^^^
|
= note: could not prove `Pin<Box<[async block@src/main.rs:186:1: 186:12]>>: CoerceUnsized<Pin<Box<(dyn futures::Future<Output = Outcome<rocket::Response<'f>, Status, rocket::Data<'g>>> + std::marker::Send + 'h)>>>`
= note: this error originates in the attribute macro `get` (in Nightly builds, run with -Z macro-backtrace for more info)Here's the function in full:
#[get("/")]
async fn index<'a>(client: &State<reqwest::Client>, token: Token) -> Result<Template, String> {
let (accounts, accounting_years) = try_join!(
get_accounts(client, &token),
get_accounting_years(client, &token)
)?;
let entries: Vec<Entry> = stream::iter(&accounting_years)
.then(|accounting_year| {
let client = &client;
let token = &token;
let year = &accounting_year.year;
async move { get_entries(client, token, &year).await }
})
.try_concat()
.await?;
Ok(Template::render(
"index",
context! { accounts: accounts, accounting_years: &accounting_years },
))
}
zjp-CN
commented
1 year ago Here's a case too: playground
#[async_trait]
trait FromText<'r> {
type Output;
async fn parse(text: &'r Text) -> Self::Output;
}
// ok
#[async_trait]
impl<'r, A> FromText<'r> for (A,)
where
A: FromText<'r>,
{
type Output = (A::Output,);
async fn parse(text: &'r Text) -> Self::Output {
let a = A::parse(text).await;
(a,)
}
}
// error: higher-ranked lifetime error, could not prove
// `Pin<Box<[async block@src/lib.rs:35:52: 39:6]>>: CoerceUnsized<Pin<Box<(dyn Future<Output = (<A as FromText<'f>>::Output, <B as FromText<'g>>::Output)> + Send + 'h)>>>`
#[async_trait]
impl<'r, A, B> FromText<'r> for (A, B)
where
A: FromText<'r>,
B: FromText<'r>,
A::Output: Send,
{
type Output = (A::Output, B::Output);
async fn parse(text: &'r Text) -> Self::Output {
let a = A::parse(text).await;
let b = B::parse(text).await;
(a, b)
}
}The solutions:
RReverser
commented
1 year ago Also ran into this, but, thanks to https://github.com/rust-lang/rust/issues/102211#issuecomment-1380167541 at least could pinpoint it to try_flatten_unordered which seems to cause the same issues. Avoiding it in favour of manual try_fold works around the problem but the error is really hard to parse.
MachariaK
commented
1 year ago I am experiencing the same problem:
let batch_size = 100;
let step = cmp::min(batch_size, members_data.len());
for i in (0..members_data.len()).step_by(step) {
let first = i;
let mut last = i + batch_size;
if last > members_data.len() {
last = members_data.len();
}
let batch = &members_data[first..last];
let member_creates = batch.into_iter().map(|member_data| {
client.member().create(
member_data.surname.to_string(),
member_data.first_name.to_string(),
member_data.gender.to_string(),
member_data.date_of_birth.into(),
member_data.kra_pin.to_string(),
member_data.nssf_number.to_string(),
member_data.national_id_number.to_string(),
member_batch::UniqueWhereParam::IdEquals(new_member_batch.clone().id),
vec![
member::title::set(Some(member_data.clone().title.unwrap_or_default())),
member::title::set(Some(member_data.clone().other_names.unwrap_or_default())),
],
)
});
//Problematic section - Compiles when this code is commented out
let new_members: Vec<member::Data> = client
._batch(member_creates)
.await
.unwrap();
assert_eq!(new_members.len(), last - first);
// End of problematic section
window
.emit_all("member-added-to-database", Payload { member_index: last })
.unwrap();
}
error: higher-ranked lifetime error
--> src\member_functions\create_member_data.rs:180:1
|
180 | #[tauri::command]
| ^^^^^^^^^^^^^^^^^
DzenanJupic
commented
1 year ago Similar issue here. Was able to boil it down to a minimal example ~without any dependencies~(with only a futures 0.3 dependency): [playground]
error: higher-ranked lifetime error
= note: could not prove `Pin<Box<[async block@src/lib.rs:21:18: 34:10]>>: CoerceUnsized<Pin<Box<(dyn futures::Future<Output = std::result::Result<Vec<()>, Error>> + std::marker::Send + 'b)>>>`Removing either the 'async_trait lifetime and using elided lifetimes as well as removing the and_then call gets rid of the error somehow. Both solutions aren't practical in my case though.
Edit:
rustc 1.68.0 (2c8cc3432 2023-03-06)
binary: rustc
commit-hash: 2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74
commit-date: 2023-03-06
host: x86_64-unknown-linux-gnu
release: 1.68.0
LLVM version: 15.0.6
oeed
commented
1 year ago I too have been running in to this a fair amount while doing refactoring to further adopt async. While attempting to find a work around I also ran in to a few "forcing query with already existing DepNode" panics like in #109655. cargo clean did resolve them.
I don't have a min repro, but a (somewhat involved) workaround for any other people who stumble upon this:
I used the assert_send function above to narrow down the exact location that caused the issue. i.e. if one .await is showing the error, try applying applying assert_send to all of the .awaits within the parent async function (and subsequent tree), it may move from the parent to one of these child .awaits.
Once you have the 'terminal' future (i.e. adding assert_send to child .awaits doesn't move the error) try moving any calculated function parameters to let bindings instead of calculating them when passed:
- some_fn(hash_map.values().map(|i| *i).collect()).await;
+ let ids = hash_map.values().map(|i| *i).collect();
+ some_fn(ids).await;I suspect it's something to do with the borrowed hash_map reference being passed to the future?
To be clear, the above example isn't a reproduction of this bug, but more a template of what my 'fix' looked like.
emersonford
commented
1 year ago Was poking at this again last night. This bug is triggered for any auto trait (#13231), e.g.
#![feature(auto_traits)]
auto trait SomeAutoTrait {}
fn main() {
let _: &dyn SomeAutoTrait = &async {
let _it = [()].iter().map(|it| None::<()>).flatten();
async {}.await;
};
}will trigger the exact same bug (playground).
I also found it curious that @danielhenrymantilla's first example in this comment produces two error message, whereas the second example produces a single error message. This plus @eholk's comment in #96865 makes me think the root of the issue is with how auto trait resolution(?) is done with HRTBs?
danielhenrymantilla
commented
1 year ago [Replying to @emersonford's post here since it seems more appropriate]
nice post @emersonford 🙏
Regarding:
What's not immediately clear to me though is why this is so specific to auto traits + generators. Even if you don't need to prove Send for the generator, don't you still need to prove that Trait is implemented for fn(&'static ()) inside of the generator interior? I.e., why does
It's specific to:
Future are erased: Rust does not have the tools to express self-referential lifetimes, so to avoid being unsound with the "made up" lifetimes that can end up populating the compiler-generated self-referential datastructure backing generators / async, the compiler straight up erases them somehow so as to prevent lifetime-specific impls from applying (since in certain scenarios this could be unsound);-> impl Trait / existential trait abstractions (that is, contrary to other traits which are just stated explicitly as part of the -> impl Trait contract, so that they can be checked within the right context, auto-traits are not enforced there and then, and instead checked later on, in a context with erased lifetimes;1.Now, the culprit triggering this is indeed the associated type, <T as Trait>::Assoc. At that point, even if the data structure itself (Foo in my example) unconditionally requires T : Trait, the trait checker will nonetheless try to prove this again, and this time with the added difficulty of having erased lifetimes, even the 'static lifetime.
Is then not to use associated types.
That is, replacing:
struct Foo<T : Trait>(
T::Assoc,
);with:
struct Foo<T : Trait<Assoc = R>, R = <T as Trait>::Assoc>(
R,
::core::marker::PhantomData<T>,
);that is, we introduce a new type R, which defaults to T::Assoc, and we also require that T::Assoc be equal to R. All in all, inference should work just as well, and basically anything else too, but for the type now containing a fully standalone R type. This will avoid the requirement to check T : Trait within an erased-lifetime scenario.
Using .flat_map(|x| x) instead of .flatten() dodges the issues, even though they are semantically equivalent. The difference is that flatten does mention <I as Iterator>::Item, whereas flat_map uses a fresh U type parameter which is only constrained to be F::Output for extra trait implementations.
Since using struct X(fn(&'static ()); dodges the issue too (since the X "type constructor" is no longer a "type with a movable lifetime within in"), I got curious about trying to formalize stuff using hand-rolled type constructors, i.e., HKTs:
trait WithLifetime<'a> { type T; }
type Feed<'a, T : WithLifetime<'a>> = T::T;
trait HKT = for<'a> WithLifetime<'a>;
type A = dyn for<'a> WithLifetime<'a, T = fn(&'static ())>;
type B = dyn for<'a> WithLifetime<'a, T = fn(&'a ())>;with macro sugar:
type A = HKT!(<'a> = fn(&'static ());
type B = HKT!(<'a> = fn(&'a ());Here, we have A and B both being fully-fledged types, but which happen to express a form of <'a>-genericity, which could be expressed in the following pseudo-code, sort to speak:
type A<'a> = fn(&'static ());
type B<'a> = fn(&'a ());And doing that, we could then replace struct Foo<T>(<T as Trait>::Assoc) where T : Trait; definition with:
/// Idea: replace `T` with `Feed<'static, T_>`, for some `T_ : <'_>` "generic generic" type.
struct Foo<T_ : ?Sized + HKT>(
<Feed<'static, T_> as Trait>::Assoc,
)
where
Feed<'static, T_> : Trait,
;With that, using Foo::<A> fails, but using Foo::<B> does not fail, even though they both amount to the same T = fn(&'static ()).
A "capturing" 'static, and thus being detected by the lifetime-eraser heuristic,B does not capture anything on its own: it's just its associated type which does, but after being queried with a fed 'static to the WithLifetime trait. And somehow that lifetime in as WithLifetime<'static> does not get erased 🤷
TheZoq2
commented
1 year ago I think I managed to reproduce the same or a similar issue in my project. This code gives the unhelpful error:
use tokio_stream::StreamExt;
async fn run_ui(tests: Vec<()>) {
loop {
let _ = tokio_stream::iter(&tests)
.then(|_| async {
})
.collect::<Vec<_>>()
.await;
}
}
#[tokio::main]
async fn main() {
tokio::spawn(run_ui(vec![]));
}[dependencies]
tokio = { version = "1.28.2", features = ["full"] }
tokio-stream = "0.1.14"Passing tests by value instead of by reference on line 4 seems to fix it
vincenzopalazzo
commented
1 year ago can you post also the unhelpful error?
TheZoq2
commented
1 year ago Ah yes, of course:
Compiling async_bug v0.1.0 (/tmp/async_bug)
error: higher-ranked lifetime error
--> src/main.rs:20:5
|
20 | tokio::spawn(run_ui(vec![]));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: could not prove `impl Future<Output = ()>: Send`
error: could not compile `async_bug` (bin "async_bug") due to previous error
hcsch
commented
1 year ago To add another example of code that causes the error:
trait Trait {
type Item;
}
impl<B, I, F> Trait for (I, F)
where
F: FnMut(I) -> B,
{
type Item = I;
}
struct Struct1<I: Trait>(I::Item);
impl<I: Trait> Struct1<I> {
fn new(i: I) -> Self {
todo!()
}
}
async fn af3() {}
async fn af2(_: &mut (), i: &()) {
let d = Struct1::new((i, |c| c));
af3().await
}
async fn af1() {
let a = Box::leak(Box::new(()));
let b = Box::leak(Box::new(()));
spawn(af2(a, b));
}
pub fn spawn<T: Send>(future: T) {
todo!()
}error: higher-ranked lifetime error
--> lib.rs:30:5
|
30 | spawn(af2(a, b));
| ^^^^^^^^^^^^^^^^
|
= note: could not prove `impl Future<Output = ()>: Send`I've hit this in a codebase of mine that uses an Iterator with map() and peekable() in an async member function, holding that across an await point. The member function is called in an async function the future of which is passed to tokio::spawn.
Somewhat unrelated: https://github.com/marxin/cvise helped a lot in getting this example down to what it is now (though you need to fix the syntax after using it, since it's designed for C).
yuri-rs
commented
1 year ago I face this issue with futures::stream::Buffered (and BufferUnordered) wrappers.
The following code reproduce the issue:
use futures::stream::StreamExt;
fn higher_ranked_lifetime_error(
) -> std::pin::Pin<std::boxed::Box<dyn std::future::Future<Output = ()> + Send>> {
Box::pin(async {
let vec_with_readers: Vec<Box<dyn std::io::Read + Send>> = Vec::new();
let _ = futures::stream::iter(
vec_with_readers
.into_iter()
.map(|_| futures::future::ready::<()>(())),
)
.buffered(1)
.count()
.await;
()
})
}while removing .buffered(1) allow the code to compile.
Playground: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=9cd1b46b17239d28184e14251910e8f7
Could anyone point me to the workaround for the issue please?
Workaround from https://github.com/rust-lang/rust/issues/102211#issuecomment-1513931928 will require changes for futures::stream::Buffered, so it is not what I'm looking for.
zertyz
commented
1 year ago I'm sure this is obvious to you, I just want to make sure it's documented somewhere:
switching
let b: Vec<&[String]> = b.chunks(2).collect();tolet b: Vec<Vec<String>> = b.chunks(2).map(|c| c.to_vec()).collect();works around the issue.
Or better yet, using @frxstrem's suggestion of replacing the closure for a function pointer and skipping copying things around unnecessarily: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=a4f501590c778b0dbe78ca620af23b4d
veeshi
commented
1 year ago Could anyone point me to the workaround for the issue please?
Workaround from #102211 (comment) will require changes for
futures::stream::Buffered, so it is not what I'm looking for.
I'm facing the same issue after introducing BufferUnordered, attempted to Pin and Box the future that is causing the error but still getting the same error.
Anyone know another workaround?
Palmik
commented
1 year ago In my case the issue was when registering a route handler with axum for foo below.
Note that foo itself compiles when used on its own, it's only when used as an axum handler that it runs into this issue.
To fix this, I had to change do_something_with_xs to take xs: Vec<X> instead of xs: &[X]. This is unintuitive to me, since it's not clear to me why the lifetimes from do_something_with_xs would be able to "escape" into the type of foo.
Another fix would be to not use buffered, but that's not acceptable in my case.
async fn foo() -> Result<()> {
let xs = vec![...];
do_something_with_xs(&xs).await
}
async fn do_something_with_xs(xs: &[X]) -> Result<()> {
// Broken version:
let ys = futures::stream::iter(xs.iter().map(|x| do_something_with_x(x)));
// Working version when using xs: Vec<X>
// let ys = futures::stream::iter(xs.into_iter().map(|x| async move { do_something_with_x(&x).await }));
// Working version when using xs: &[X] without buffered
// let ys = futures::stream::iter(xs.iter()).then(|x| do_something_with_x(x));
tokio::pin!(stream);
while let Some(y) = ys.next().await {
// Do something with y
}
}
// You can keep the reference here
async fn do_something_with_x(x: &X) -> Result<Y> {
// ...
}
UVUUUVUU
commented
1 year ago i have a same problem.
pub async fn select_paginate() {
match get_sql_db_conn().await.unwrap() {
DBPool::AsyncPgPool(pool) => {
let mut conn = pool.get().await.unwrap();
conn.transaction::<(), diesel::result::Error, _>(|conn_mutex| async move {
let sql = "SELECT * FROM tableone WHERE id = 1".to_string();
diesel::sql_query(&sql).execute(conn_mutex).await.unwrap();
let mut db_query = tableone.into_boxed();
db_query = db_query.filter(title.eq("shenhu"));
db_query = db_query.limit(1).offset(1);
let result = db_query.paginate(1, 1).load_and_total::<Tableone>(conn_mutex).await.unwrap();
println!("{:?}", result);
Ok(())
}.scope_boxed())
.await;
}
_ => {}
};
}the error:
higher-ranked lifetime error
could not prove `[async block@src/utils/select_async.rs:84:75: 93:18]: std::marker::Send`
Running stable 1.64.0.
Unfortunately this happened in a complex bit of code that I can't easily reduce :(
The future returned by
handle_service.call(connection)is definitelySend, and I can work around the failure by boxing it: