enso-org / enso

Enso Analytics is a self-service data prep and analysis platform designed for data teams.
https://ensoanalytics.com
Apache License 2.0
7.38k stars 323 forks source link

Ensure `Value_Type.Byte` behaves consistently between in-memory and Database #7888

Open radeusgd opened 1 year ago

radeusgd commented 1 year ago

We have a Value_Type.Byte which is a special type used to represent small 8-bit integers.

Most database systems will treat such type as an unsigned byte with the range of values being 0-255. However, our in-memory backend currently relies on Java's defaults where all numeric types are signed and the byte values are actually -128 to 127. This is not consistent with databases.

radeusgd commented 1 year ago

I think this task is kind of blocked by waiting for new database integrations.

That is because our current two backends do not really support a Byte type:

So I think it will only matter once we actually have a Database that can store 8-bit integers, e.g. SQL Server that has tinyint.


It may make sense to tackle the task earlier to ensure we don't introduce a breaking change later (changing the range on Value_Type.Byte from current -128 to 127 to 0-255), though.

radeusgd commented 1 year ago

Just changing the 'working range' should actually be trivial with the current implementation that stores everything as a long. It can easily be done in probably less than half a day. So we may consider doing this change right away and then doing a separate integration once we have a Byte supporting DB backend.

jdunkerley commented 10 months ago

Need to review that round trip is possible.

Should get the value from a byte column as a 0-255 (stored as a long behind the scenes). Need to be able to write back to same DB tech with same type.

jdunkerley commented 10 months ago

It is on hold as no backend currently supported which has a byte type.