pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust
https://docs.pola.rs
Other
29.59k stars 1.89k forks source link

SchemaError for Non-Exiting dtype on Concat #17300

Closed starzar closed 1 month ago

starzar commented 3 months ago

Checks

Reproducible example

dfAnalysis_total = pl.DataFrame()
df_1 = 18-06-2024(CotFinancials)_dfAnalysis_cur
df_2 = 25-06-2024(CotFinancials)_dfAnalysis_cur
df_list =  [df_1,df_2]

for df_cur in range(df_list):
    # SchemaError: type; Int64 is incompatible with expected type Float64
    # But I64 does not exist in any of the concatenated dataframes
    dfAnalysis_total = pl.concat([dfAnalysis_total,df_cur],how="diagonal_relaxed")

Log output

Traceback (most recent call last):
  File "C:\Users\User_0\Documents\Code\Python\NseScraping\Others\CotReport\py\cotAnalysis_financials.py", line 219, in txt_to_df
    cotSoloAnalysis(cot_df_total.with_row_index(),fileName)
  File "C:\Users\User_0\Documents\Code\Python\NseScraping\Others\CotReport\py\cotAnalysis_financials.py", line 437, in cotSoloAnalysis
    dfAnalysis_total = pl.concat(items = [dfAnalysis_total,dfAnalysis_cur],how="diagonal")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User_0\AppData\Local\Programs\Python\Python312\Lib\site-packages\polars\functions\eager.py", line 196, in concat
    out = wrap_df(plr.concat_df_diagonal(elems))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.SchemaError: type Int64 is incompatible with expected type Float64

Issue description

https://drive.google.com/file/d/1x182gU8SwwcuHtXGom8JsGC1kuIUC_Br/view?usp=sharing I64 does not exist in any of the concatenated dataframes but polars gives " SchemaError: type; Int64 is incompatible with expected type Float64"

Expected behavior

I64 should not shown as a dtype at all.

Installed versions

``` polars 0.20.31 ```
ritchie46 commented 3 months ago

Can you create a reproducible example?

starzar commented 3 months ago

Error starts when concatenating 2nd row onwards - shape: (1, 5) ┌───────────────────┬───────────────────┬───────────────────┬───────────────────┬──────────────────┐ │ Dealer ┆ AssetManager ┆ LeveragedFunds ┆ OtherReportables ┆ NonReportables │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] │ ╞═══════════════════╪═══════════════════╪═══════════════════╪═══════════════════╪══════════════════╡ │ {{5637.0,10503.0, ┆ {{761.0,1541.0,14 ┆ {{813.0,22040.0,2 ┆ {{2763.0,13974.0, ┆ {{690.0,1812.0,3 │ │ 761.0},{11.3,21.0 ┆ 59.0},{1.5,3.1,2. ┆ 6343.0},{1.6,44.0 ┆ 6392.0},{5.5,27.9 ┆ 06.0},{1.4,3.6,0 │ │ ,1.5},{166.0,-470 ┆ 9},{761.0,-434.0, ┆ 99998,52.700001}, ┆ ,12.8},{89.0,774. ┆ .6},{159.0,-179. │ │ 4.0,761.0},{4.0,5 ┆ 655.0},{15.0,11.0 ┆ {221.0,3443.0,739 ┆ 0,382.0},{0.0,0.0 ┆ 0,37.0},{0.0,0.0 │ │ .0,15.0}} ┆ ,7.0}} ┆ 9.0},{97.0,25.0,8 ┆ ,0.0}} ┆ ,0.0}} │ │ ┆ ┆ .0}} ┆ ┆ │ └───────────────────┴───────────────────┴───────────────────┴───────────────────┴──────────────────┘ cot_df_cur shape: (1, 5) ┌───────────────────┬───────────────────┬───────────────────┬───────────────────┬──────────────────┐ │ Dealer ┆ AssetManager ┆ LeveragedFunds ┆ OtherReportables ┆ NonReportables │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] │ ╞═══════════════════╪═══════════════════╪═══════════════════╪═══════════════════╪══════════════════╡ │ {{0.0,3701.0,0.0} ┆ {{0.0,0.0,0.0},{0 ┆ {{0.0,902.0,17450 ┆ {{215.0,21676.0,3 ┆ {{0.0,2517.0,310 │ │ ,{0.0,14.6,0.0},{ ┆ .0,0.0,0.0},{0.0, ┆ .0},{0.0,3.6,68.9 ┆ 634.0},{0.8,85.59 ┆ .0},{0.0,9.9,1.2 │ │ 0.0,1084.0,0.0},{ ┆ 0.0,0.0},{0.0,0.0 ┆ 00002},{0.0,723.0 ┆ 9998,14.4},{184.0 ┆ },{0.0,172.0,-2. │ │ 0.0,0.0,0.0}} ┆ ,0.0}} ┆ ,2855.0},{7.0,4.0 ┆ ,1879.0,-1163.0}, ┆ 0},{0.0,0.0,0.0} │ │ ┆ ┆ ,123.0}} ┆ {19.0,0.0,0.0}} ┆ } │ └───────────────────┴───────────────────┴───────────────────┴───────────────────┴──────────────────┘ cot_df_curtype Int64 is incompatible with expected type Float64

shape: (1, 5) ┌───────────────────┬───────────────────┬───────────────────┬───────────────────┬──────────────────┐ │ Dealer ┆ AssetManager ┆ LeveragedFunds ┆ OtherReportables ┆ NonReportables │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] │ ╞═══════════════════╪═══════════════════╪═══════════════════╪═══════════════════╪══════════════════╡ │ {{127.0,22378.0,0 ┆ {{0.0,13428.0,464 ┆ {{529.0,17397.0,1 ┆ {{444.0,4651.0,17 ┆ {{1.0,3691.0,174 │ │ .0},{0.3,55.59999 ┆ .0},{0.0,33.29999 ┆ 2994.0},{1.3,43.2 ┆ 15.0},{1.1,11.6,4 ┆ 2.0},{0.0,9.2,4. │ │ 8,0.0},{127.0,208 ┆ 9,1.2},{-872.0,-2 ┆ 00001,32.299999}, ┆ .3},{-914.0,2697. ┆ 3},{-28.0,-170.0 │ │ 91.0,-872.0},{0.0 ┆ 99.0,69.0},{41.0, ┆ {-99.0,1470.0,-16 ┆ 0,-686.0},{0.0,0. ┆ ,-448.0},{0.0,0. │ │ ,22.0,41.0}} ┆ 11.0,14.0}} ┆ 001.0},{9.0,0.0,0 ┆ 0,0.0}} ┆ 0,0.0}} │ │ ┆ ┆ .0}} ┆ ┆ │ └───────────────────┴───────────────────┴───────────────────┴───────────────────┴──────────────────┘ type Int64 is incompatible with expected type Float64

cmdlineluser commented 3 months ago

@starzar You'll need to provide a full code example in order for other people to run it and reproduce the problem.