Non-deterministic Ranges in Inference Synthesis Causing Inconsistent State Across Nodes

Summary

In the inference_synthesis package, the inference synthesis process is utilizing non-deterministic ranges. This non-deterministic behavior can lead to inconsistent state across different nodes, particularly for any data saved from the inference synthesis. The inconsistency can cause significant issues in the system's overall functionality and reliability.

Vulnerability Detail

In the inference_synthesis package, the inference synthesis process is utilizing non-deterministic ranges. This non-deterministic behavior can lead to inconsistent state across different nodes, particularly for any data saved from the inference synthesis. The inconsistency can cause significant issues in the system's overall functionality and reliability.

The code section in question involves creating deep copies of various maps and structs within the SynthPalette struct. While the cloning function is properly creating deep copies, the inference synthesis process itself is not deterministic, which leads to the described issue.

Impact

This issue can cause multiple nodes in a distributed system to have differing states due to the non-deterministic ranges used during inference synthesis. This can result in:

Inconsistent inference and forecast results across nodes.
Difficulty in debugging and maintaining the system due to unpredictable state changes.
Potential data corruption or loss of data integrity, which can affect overall system performance and reliability.
Issues in consensus mechanisms if nodes cannot agree on a single state.

Code Snippet

synth_palette_bootstrap.go#L58-L82

func (p SynthPalette) Clone() SynthPalette {
    inferenceByWorker := make(map[Worker]*emissionstypes.Inference, len(p.InferenceByWorker))
    for k, v := range p.InferenceByWorker {
        inferenceCopy := *v
        inferenceByWorker[k] = &inferenceCopy
    }
    forecastByWorker := make(map[Worker]*emissionstypes.Forecast, len(p.ForecastByWorker))
    for k, v := range p.ForecastByWorker {
        forecastCopy := *v
        forecastByWorker[k] = &forecastCopy
    }
    forecastImpliedInferenceByWorker := make(map[Worker]*emissionstypes.Inference, len(p.ForecastImpliedInferenceByWorker))
    for k, v := range p.ForecastImpliedInferenceByWorker {
        inferenceCopy := *v
        forecastImpliedInferenceByWorker[k] = &inferenceCopy
    }
    infererRegrets := make(map[Worker]*StatefulRegret, len(p.InfererRegrets))
    for k, v := range p.InfererRegrets {
        regretCopy := *v
        infererRegrets[k] = &regretCopy
    }
    forecasterRegrets := make(map[Worker]*StatefulRegret, len(p.ForecasterRegrets))
    for k, v := range p.ForecasterRegrets {
        regretCopy := *v
        forecasterRegrets[k] = &regretCopy
    }

Tool used

Manual Review

Recommendation

Review the inference synthesis process and identify the sources of non-determinism.
Ensure that all ranges and related calculations are deterministic, i.e., they produce the same output given the same input across all nodes.

sherlock-admin3 commented 1 month ago

1 comment(s) were left on this issue during the judging contest.

0xmystery commented:

Map is non-deterministic

sherlock-admin2 commented 1 month ago

The protocol team fixed this issue in the following PRs/commits: https://github.com/allora-network/allora-chain/pull/408

sherlock-audit / 2024-06-allora-judging

defsec - Non-deterministic Ranges in Inference Synthesis Causing Inconsistent State Across Nodes #38