[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of Polars.
Reproducible example
df: pl.LazyFrame = pl.scan_parquet('s3://...')
df.collect() # fine
# wait an hour (or whatever length AWS access tokens go stale)
df.collect() # fails
Log output
No response
Issue description
I'm having issues with s3 lazy frames. It seems like when I use them immediately after creation and collect()ing it works fine. But I have a long running process where the lazy frame is defined, hours of computation is done, and then the lazy frame gets collected again, and it fails.
Our AWS credentials go stale after ~1hr, so if the computation takes longer than that it seems to fail. As far as I can tell from reading the cred provider code, the tokens are computed when the frame is created, and not when it's collectd
Expected behavior
Tokens are evaluated on collect, not on initialization of the LazyFrame
Checks
Reproducible example
Log output
No response
Issue description
I'm having issues with s3 lazy frames. It seems like when I use them immediately after creation and
collect()
ing it works fine. But I have a long running process where the lazy frame is defined, hours of computation is done, and then the lazy frame getscollect
ed again, and it fails.Our AWS credentials go stale after ~1hr, so if the computation takes longer than that it seems to fail. As far as I can tell from reading the cred provider code, the tokens are computed when the frame is created, and not when it's
collect
dExpected behavior
Tokens are evaluated on
collect
, not on initialization of theLazyFrame
Installed versions