Fix BCD / decimal type converion to use correct size and handle negat…ive values.

DataZooDE / erpl

ERPL is a DuckDB extension to integrate Enterprise Data in your Data Science and ML pipelines within minutes! ERPL connects DuckDB to SAP ecosystem via standard interfaces.

https://erpl.io

Other

31 stars 1 forks source link

Fix BCD / decimal type converion to use correct size and handle negat…ive values. #8

Closed jrosskopf closed 9 months ago

jrosskopf commented 9 months ago

Try to fix #5. By changing the treatment of BCD values. We were blindly using string representation of BCD reveived from the RFC framework and default DuckDB cast. This pull requests adds

Proper fix of string lenght and termination
Handling of the negativ number suffix and converting it to a prefix/sign at the begining.

Hope it helps.

gustavoleo commented 9 months ago

super

gustavoleo commented 9 months ago

I couldn't succeed on my lab! How about you Simon?

sipemu commented 9 months ago

@gustavoleo, I also did some more tests regarding the decimal bug. It seems to be fine now. I build some demos for modern data stacks when I have free time from the main business.

gustavoleo commented 9 months ago

@sipemu Any aditional logs needed?

con.sql(""" SELECT MANDT, BUKRS, KZKRS, BLART, BUDAT, MONAT, KURSF FROM SAP_READ_TABLE('BKPF') LIMIT 5 """)

-----(newtux) tony@lainoborgo:~/.duckdb/extensions$ ls v0.10.0

InvalidInputException: Invalid Input Error: Failed to cast value: Could not convert string " 1.24140-" to DECIMAL(9,5)

gustavoleo commented 8 months ago

From BKPF if we Try to write rto Parquet go got a error in some Date fields like AEDAT [BKPF.zip] XTRACT_BKPF.zip (https://github.com/DataZooDE/erpl/files/14510320/BKPF.zip)