jerrinot
commented
1 year ago hello @awfm9: thanks for a nicely described feature request!
what arithmetic operations do you see as the absolute must? something that would make the difference for you, so you would say: "Ok, this is way better than just wrapping hashes in a binary column on my own. This long256 type is really making my life easier!"?
and one more question: what do you currently use for visualization? the only visualization questdb offers out of the box is in the web console. is long256 charting support what you had in mind for visualization? or do you mean support in Grafana and similar tools? Would it help if we send long256 values encoded as the Postgres Numeric type?
Is your feature request related to a problem?
QuestDB claims on its website to support 256-bit unsigned integers. Upon consultation with the team, it became clear, though, that QuestDB does not truly support 256-bit integers.
On closer inspection of the documentation, it turns out that the
Long256type is merely the hex-encoded binary value of 256-bit unsigned integers, which follows the same semantics as other binary data:https://questdb.io/docs/reference/api/ilp/columnset-types#long256
There is no support for basic arithmetic operations, or for visualization of the integer values. With such limitations, the support for 256-bit integers is nothing more than semantic sugar at this point.
Describe the solution you'd like.
In the blockchain space, the Ethereum Virtual Machine (EVM) operates with 256-bit integers by default. As such, running precise simulations on top of collected blockchain data would require the storage and retrieval of full 256-bit values. For this basic requirement, the current
Long256support is sufficient, but so is the feature set of any other time series database that allows storage of binary values. Hex-encoding the binary format of a 256-bit integer is literally just a few extra lines of code.In order to draw proper conclusions from the simulations, it's however also important to visualize the data. In fact, the ability to do basic arithmetic operations on the results would further improve the value of the database by allowing even deeper insights to be glanced from the graphs. Without this feature, the database is nothing but a dumb storage layer.
Describe alternatives you've considered.
We are currently running our simulations on top of InfluxDB Cloud. Initially, we used 64-bit float values to represent the data. This lead to compounding off-by-one errors. Barely noticeable after a few months of simulation, they exponentially increased to a point where the results were unusable after a year of block data.
As a solution, we started hex-encoding the full 256-bit integers and storing them as strings in the database. This, however, broke our ability to visualize the data with InfluxDB's dashboard.
Upon doing some research, we found that QuestDB had 256-bit integer support. However, it turns out that it amounts to the exact same thing as we are already manually doing for InfluxDB.
Additional context.
All applications built on top of the EVM use 256-bit integers. This equates to 99% of relevant Decentralized Finance (DeFi) applications. While many of the users do currently not care about precision of values, I would argue that the need will grow significantly as the sector professionalizes further. Having a time series database with full support for 256-bit integers would offer an excellent platform for accurate data analytics for blockchain applications at a granular level.