search

pola-rs / nodejs-polars

nodejs front-end of polars
https://pola-rs.github.io/nodejs-polars/
MIT License
437 stars 44 forks source link

Polars scan parquet not detecting additional columns in some partitions #283

Closed Sankartk closed 1 month ago

Sankartk commented 1 month ago

Have you tried latest version of polars?

Yes

What version of polars are you using?

1.19.0

What operating system are you using polars on?

WINDOWS 11

Describe your bug.

I'm using polars scan parquet method to read parquet files from a directory path which has partitions. For some partitions, the file has new columns added where polars is not scanning those. I also tried allowing missing columns equals true even that didn't fix the error

Bidek56 commented 1 month ago

@Sankartk Do you have a MRE? Do you experience this problem using Python-polars? Have you tried to look for a solution in SO? Thx

Sankartk commented 1 month ago

Yes python polars. I tried to look at the solution where they asked to use allow_missing_columns= True field on using scan_parquet method but that didn't help as well

Bidek56 commented 1 month ago

This is a Python Polars issue, not a Nodejs issue.