search

hashgraph / hedera-services

Crypto, token, consensus, file, and smart contract services for the Hedera public ledger
Apache License 2.0
265 stars 119 forks source link

Investigate mutability of state during app initialization with reconnect trigger #14116

Open poulok opened 2 days ago

poulok commented 2 days ago

The services team ran into unexplained behavior during an experiment. The code attempted to perform a data migration on application initialization with the InitTrigger.RECONNECT trigger and got this exception:

2024-07-01 16:08:28.990 INFO  98   V053AddressBookSchema - Migrated 4 nodes from address book
2024-07-01 16:08:28.990 FATAL 385  Hedera - Critical failure during initialization
com.swirlds.base.state.MutabilityException: This operation is not permitted on immutable leaves
    at com.swirlds.virtualmap.internal.cache.VirtualNodeCache.throwIfLeafImmutable(VirtualNodeCache.java:1555) ~[swirlds-virtualmap-0.52.0.jar:?]
    at com.swirlds.virtualmap.internal.cache.VirtualNodeCache.putLeaf(VirtualNodeCache.java:546) ~[swirlds-virtualmap-0.52.0.jar:?]
    at com.swirlds.virtualmap.internal.cache.VirtualNodeCache.lookupLeafByKey(VirtualNodeCache.java:675) ~[swirlds-virtualmap-0.52.0.jar:?]
    at com.swirlds.virtualmap.internal.merkle.RecordAccessorImpl.findLeafRecord(RecordAccessorImpl.java:132) ~[swirlds-virtualmap-0.52.0.jar:?]
    at com.swirlds.virtualmap.internal.merkle.VirtualRootNode.getForModify(VirtualRootNode.java:815) ~[swirlds-virtualmap-0.52.0.jar:?]
    at com.swirlds.virtualmap.VirtualMap.getForModify(VirtualMap.java:439) ~[swirlds-virtualmap-0.52.0.jar:?]
    at com.swirlds.platform.state.merkle.disk.OnDiskWritableKVState.putIntoDataSource(OnDiskWritableKVState.java:111) ~[swirlds-platform-core-0.52.0.jar:?]
    at com.swirlds.platform.state.spi.WritableKVStateBase.commit(WritableKVStateBase.java:55) ~[swirlds-platform-core-0.52.0.jar:?]
    at com.swirlds.platform.state.merkle.disk.OnDiskWritableKVState.commit(OnDiskWritableKVState.java:146) ~[swirlds-platform-core-0.52.0.jar:?]
    at com.hedera.node.app.state.merkle.MerkleHederaState$MerkleWritableStates.commit(MerkleHederaState.java:767) ~[app-0.52.0.jar:?]
    at com.hedera.node.app.state.merkle.MerkleSchemaRegistry.migrate(MerkleSchemaRegistry.java:254) ~[app-0.52.0.jar:?]
    at com.hedera.node.app.services.OrderedServiceMigrator.lambda$doMigrations$2(OrderedServiceMigrator.java:116) ~[app-0.52.0.jar:?]

It makes total sense that the reconnect state would be immutable during this initialization. The part that requires investigation is why an exception was not thrown here instead - a call that was made just a few lines before the call that caused the exception.

This ticket is complete when the root cause has been identified. If there is any correction needed, a ticket should be written capturing the work required.