search

vaexio / vaex

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
https://vaex.io
MIT License
8.25k stars 590 forks source link

[FEATURE-REQUEST] 128 bit data type support #2141

Open lehoai opened 2 years ago

lehoai commented 2 years ago

Description We are developing a system related to the blockchain, and the data type of 64 bits is not enough. It will be better if vaex support 128bit data type.

Is your feature request related to a problem? Please describe. No

Additional context

maartenbreddels commented 2 years ago

What kind of operations do you need to perform on it?

On Fri, 29 Jul 2022 at 12:31, Le Hoai @.***> wrote:

Description We are developing a system related to the blockchain, and the data type of 64 bits is not enough. It will be better if vaex support 128bit data type.

Is your feature request related to a problem? Please describe. No

Additional context

— Reply to this email directly, view it on GitHub https://github.com/vaexio/vaex/issues/2141, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANPEPMX2MLPJGFHTHZ7LNDVWOXIBANCNFSM55AJLB2A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Maarten Breddels Software engineer / consultant / data scientist Python / C++ / Javascript / Jupyter www.maartenbreddels.com / vaex.io @.*** +31 6 2464 0838 <+31+6+24640838> [image: Twitter] https://twitter.com/maartenbreddels[image: Github] https://github.com/maartenbreddels[image: LinkedIn] https://linkedin.com/in/maartenbreddels[image: Skype]

lehoai commented 2 years ago

@maartenbreddels We are trying to import data that have 22 digits but vaex doesn't support it.

maartenbreddels commented 2 years ago

Integers I assume right? But you also need so perform operations in the numbers right?

lehoai commented 2 years ago

@maartenbreddels thank you for your reply. Yeah, it is int number, i need to perform + - * / on it.

bls-lehoai commented 1 year ago

@maartenbreddels In other word, could you support Decimal type?

maartenbreddels commented 1 year ago

We could support it via Pyarrow I think. Does Pyarrow have what you need?

bls-lehoai commented 1 year ago

@maartenbreddels we are trying to import data from CSV which has a data number with 22 digits. Could you give me an example of how to use Pyarrow to import? I've read this tutorial https://vaex.readthedocs.io/en/latest/guides/arrow.html

but can't imagine how to import.

============== add

and i wanna store data in an "arrow" file (or hdf5 if possible) with the data type decimal128 to calculate on it. The problem with the float64 data type is the calculating result is incorrect when the number is larger or smaller

bls-lehoai commented 1 year ago

@maartenbreddels or maybe I misunderstood your words. it doesn't support decimal128 yet and you can support it in the next release?

JovanVeljanoski commented 1 year ago

Yeah we need to add/support pa.Decimal128 probably.

@lehoai if you use vaex.open to read a CSV file, you are already using arrow. Otherwise you can also use vaex.from_csv_arrow. But we need to support the right dtypes that you care about.

bls-lehoai commented 1 year ago

@JovanVeljanoski Thanks for your answer. Please add pa.Decimal128 type. It helps me a lot^^. I am waiting for it 👍