pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust
https://docs.pola.rs
Other
30.52k stars 1.98k forks source link

Issue with lazy frames reading from s3 in long running process #19933

Open kfreisen opened 2 days ago

kfreisen commented 2 days ago

Checks

Reproducible example

df: pl.LazyFrame = pl.scan_parquet('s3://...')
df.collect() # fine

# wait an hour (or whatever length AWS access tokens go stale)
df.collect() # fails

Log output

No response

Issue description

I'm having issues with s3 lazy frames. It seems like when I use them immediately after creation and collect()ing it works fine. But I have a long running process where the lazy frame is defined, hours of computation is done, and then the lazy frame gets collected again, and it fails.

Our AWS credentials go stale after ~1hr, so if the computation takes longer than that it seems to fail. As far as I can tell from reading the cred provider code, the tokens are computed when the frame is created, and not when it's collectd

Expected behavior

Tokens are evaluated on collect, not on initialization of the LazyFrame

Installed versions

``` >>> pl.show_versions() --------Version info--------- Polars: 1.13.0 Index type: UInt32 Platform: macOS-14.4.1-arm64-arm-64bit Python: 3.9.17 (main, Jun 6 2023, 14:33:55) [Clang 14.0.3 (clang-1403.0.22.14.1)] LTS CPU: False ----Optional dependencies---- adbc_driver_manager altair cloudpickle 3.1.0 connectorx deltalake fastexcel fsspec 2023.12.2 gevent great_tables matplotlib 3.9.2 nest_asyncio numpy 1.26.4 openpyxl pandas 2.2.3 pyarrow 17.0.0 pydantic 2.9.2 pyiceberg 0.7.1 sqlalchemy 2.0.36 torch 2.0.1 xlsx2csv xlsxwriter ```
ritchie46 commented 1 day ago

@nameexhaustion it seems that we don't re-auth?