cube2222 / octosql

OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
Mozilla Public License 2.0
4.74k stars 201 forks source link

Add int64 support #331

Closed tjungblu closed 1 month ago

tjungblu commented 2 months ago

fixes #330

@cube2222 I hope you're still alive. I'm seriously bitten by the lack of int64 for byte sizes and other large aggregations, so it would be incredible if we could add this one way or the other.

cube2222 commented 2 months ago

Hey @tjungblu, alive and well! Just didn't have time for OctoSQL for a while now.

Since you're hitting this issue, are you on a 32-bit system? My assumption with OctoSQL was mostly building for 64-bit systems, where int would be 64-bit anyways.

I have fairly intentionally avoided different bit type variants in the octosql type system to avoid the complexity. What would you think, instead, about changing all the places that are problematic where simple int is used, to just use int64? Basically, what you did in your PR, but instead of creating new structures, modifying the existing ones to just always use int64. Would that work?

tjungblu commented 2 months ago

You're correct, int is 64 bits on a 64 bit system:

$ octosql "SELECT 2147483647+2147483647, 9223372036854775807+9223372036854775807"
+------------+-------+
|   col_0    | col_1 |
+------------+-------+
| 4294967294 |    -2 |
+------------+-------+

On a raspberry pi armv7l, which is 32 bit it's unfortunately not:

$ octosql "SELECT 2147483647+2147483647, 9223372036854775807+9223372036854775807"
+-------+-------+
| col_0 | col_1 |
+-------+-------+
|    -2 |    -2 |
+-------+-------+

I realize there's no official 32 bit build of octosql, but it's somewhat inconvenient you have to be aware of your processor to get the correct result :roll_eyes:

but instead of creating new structures, modifying the existing ones to just always use int64. Would that work?

let me try! I'll send you another follow-up PR, we can decide what to make of it.

Thanks @cube2222 :)