Netflix / hollow

Hollow is a java library and toolset for disseminating in-memory datasets from a single producer to many consumers for high performance read-only access.
Apache License 2.0
1.2k stars 216 forks source link

Type count validator on first cycle after startup does not even initiate the validator #673

Open arpitmalhotra opened 7 months ago

arpitmalhotra commented 7 months ago

We have a type count validator that simply validates all type counts have not deviated. However on first cycle of startup, even though the type counts deviated, the validator was not even kicked off. See below hasChangedSinceLastCycle returns false. We do restore at startup

if (writeEngine.hasChangedSinceLastCycle()) {
                boolean schemaChangedFromPriorVersion = readStates.hasCurrent() &&
                        !writeEngine.hasIdenticalSchemas(readStates.current().getStateEngine());
                updateHeaderTags(writeEngine, toVersion, schemaChangedFromPriorVersion);

                // 3a. Publish, run checks & validation, then announce new state consumers
                publish(listeners, toVersion, artifacts);

                ReadStateHelper candidate = readStates.roundtrip(toVersion);
                cycleStatus.readState(candidate.pending());
                candidate = doIntegrityCheck ? 
                        checkIntegrity(listeners, candidate, artifacts, schemaChangedFromPriorVersion) :
                            noIntegrityCheck(candidate, artifacts);

                try {
                    validate(listeners, candidate.pending());
clay-mayers commented 7 months ago

hasChangeSinceLastCycle() checks across all types if the ordinals present for each type are different from before populate() ran compared to after. If it returns false, the populator passed into runcycle() produced an identical data state to what was restored. I don't know exactly what you mean by "type count" but if it deviated and you want validation to be triggered, the populator passed to runcycle() needs to create a change in which ordinals are present as compared to the restored data state for at least one of the types.

At least that's my reading of the code.