RFC: Change value type backing to 128-bit integers

[cjdsellers]

The Nautilus value types Price, Money, and Quantity are currently backed by 64-bit integer raw values, with a maximum precision of 9 decimals enforced through validation (see table below). This design provides memory efficiency and high performance, using simple fixed-point arithmetic to represent actual values. (Integers are more CPU-native than string or decimal types, and are highly efficient for comparisons.)

Limitations of the current 64-bit specification:

• 9 decimals of precision cannot fully represent all fractional units for certain digital assets (cryptocurrencies).
• The allowable minimum and maximum value range is limiting for some use cases.
• For Quantity, larger time frame bar volumes cannot be adequately represented.

To address these issues, increasing the raw integer width from 64 bits to 128 bits will significantly expand the precision and allowable range.

Current specification (64-bit integer backing):	Type	Raw backing	Max precision	Min value	Max value
Price	i64	9	-9,223,372,036	9,223,372,036
Money	i64	9	-9,223,372,036	9,223,372,036
Quantity	u64	9	0	18,446,744,073

Proposed specification (128-bit integer backing):	Type	Raw backing	Max precision	Min value	Max value
Price	i128	18	-170,141,183,460	170,141,183,460
Money	i128	18	-170,141,183,460	170,141,183,460
Quantity	u128	18	0	340,282,366,920

Pros of 128-bit backing:

• 18 decimals of precision can represent practically all fractional units, making it suitable for most cryptocurrencies.
• Increased headroom resolves limitations in representing large bar volumes and extreme price values.

Cons of 128-bit backing:

• Increased in-memory footprint (nearly doubling for some types such as QuoteTick).
• While Parquet efficiently compresses data, there will still be a modest increase in storage size.
• Breaking change to specification and API requiring catalogs to be re-written to upgrade to the latest version.

Other solutions, such as introducing an additional scaling factor field and using arithmetic for more flexible precisions, were considered. However, these approaches significantly increased complexity and the likelihood of bugs, as raw values are directly accessed in many parts of the codebase.

Mitigations

For the in-memory cache, the number of 128-bit fields across all objects is relatively small, and for data is further constrained by deques with a defined maxlen.

For the in-memory footprint of data during large backtests, this is mostly alleviated by streaming from the Parquet data catalog where only a limited amount of data is held in memory at a time. Users who are using the BacktestEngine directly will be most affected.

The breaking change to catalogs is a temporary inconvenience. Users can decide whether to upgrade to nautilus_trader versions with 128-bit backing or choose the timing of their upgrade to align with their operational needs. Alternatively, they can compile from source with the existing 64-bit backing.

Implementation

To provide flexibility, @twitu is implementing a solution where a high-precision feature flag will control whether these values are backed by 64-bit or 128-bit integers. This allows users to:

• Retain the current 64-bit specification for reduced memory usage.
• Opt-in to 128-bit backing to overcome precision and range limitations.

2072

Final comments

While the increased precision and range may be unnecessary or even excessive for traditional financial assets, the limitations are far more severe for crypto users. Traditional users may appreciate the lower memory footprint as a "nice to have," but crypto users are constrained by the 9-decimal precision cap and a more restricted range.

It's expected that high-precision will quickly become the default, as the advantages outweigh the trade-offs for most users.

We're open to feedback and suggestions on the above.

[davidsblom]

Quick question on the fact that existing catalogs need to be upgraded. Will there be a convenience tool to convert an existing catalog from 64-bit to 128-bit?