PanicException: index: 8449 out of bounds for len: 1 when using scan csv with schema and include_file_paths

[djouallah]

Checks

• [X] I have checked that this issue has not already been reported.
• [X] I have confirmed this bug exists on the latest version of Polars.

Reproducible example

public notebook https://colab.research.google.com/drive/1XpcKetkpN86XXx-rEhPzdL9cEAHevtEH?usp=sharing

Log output

PanicException                            Traceback (most recent call last)
<ipython-input-8-e6e2c1b6eee6> in <cell line: 6>()
      4 list_files=[os.path.basename(x) for x in glob.glob(Source+'*.CSV')]
      5 files_to_upload_full_Path = [Source + i for i in list_files][:total_files]
----> 6 polars_clean_csv(files_to_upload_full_Path)

1 frames
/usr/local/lib/python3.10/dist-packages/polars/lazyframe/frame.py in collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, no_optimization, streaming, engine, background, _eager, **_kwargs)
   2025         # Only for testing purposes
   2026         callback = _kwargs.get("post_opt_callback", callback)
-> 2027         return wrap_df(ldf.collect(callback))
   2028 
   2029     @overload

PanicException: index: 8449 out of bounds for len: 1

Issue description

it should work

Expected behavior

it should work

Installed versions

1.5.0

[djouallah]

@ritchie46 with 1.6 i have a different bug

   2032         # Only for testing purposes
   2033         callback = _kwargs.get("post_opt_callback", callback)
-> 2034         return wrap_df(ldf.collect(callback))
   2035 
   2036     @overload

PanicException: index out of bounds: the len is 52 but the index is 52

[nameexhaustion]

@djouallah Can you provide a backtrace (set RUST_BACKTRACE=1 in the environment before importing polars)?

[djouallah]

@nameexhaustion i did provide a reproduced notebook in colab, just run it there ?

[nameexhaustion]

I'm not able to get the backtrace from the colab environment - could you try and make a small script that reproduces locally?

[djouallah]

sorry for that, even locally, I could not get a backtrace working

[ritchie46]

@nameexhaustion it is related to include_file_paths.

I also see that include_file_paths doesn't respect the projection pushdown:

import polars as pl

q = pl.scan_csv("a,b,c\na1,b1,c1".encode(), include_file_paths="path_name").select(["a", "b"])

print(q.collect())

shape: (1, 3)
┌─────┬─────┬───────────┐
│ a   ┆ b   ┆ path_name │
│ --- ┆ --- ┆ ---       │
│ str ┆ str ┆ str       │
╞═════╪═════╪═══════════╡
│ a1  ┆ b1  ┆ in-mem    │
└─────┴─────┴───────────┘

[nameexhaustion]

The 2nd panic issue will be tracked at https://github.com/pola-rs/polars/issues/19397

[djouallah]

wonderful, it is fixed now !!!