The heartbeat datum at 18:00 UTC on 6 March 2024 did not make it on-chain. A datum was published manually at 18:14 UTC after the anomaly was spotted through network monitoring.
Status
Under Review
Assessment
The validator node caused the websocket connection to the COOP publishing server to be dropped. The reliability of the websocket connection and potential changes to the publishing mechanism remain under investigation.
Additional Notes
While not directly related to INCIDENT 019 we know that the reliability of the websocket connection between the validator and publishing node can impact publication. How a heartbeat is requested and subsequent monitoring of its successful publication continues to be looked at as the Orcfax network is incrementally upgraded.
We are investigating:
More advanced connection management by the Orcfax federated validator.
More resilient publishing workflows able to request a new datum if
publication isn't successful.
Trigger
Date
2024-03-06
Summary
The heartbeat datum at 18:00 UTC on 6 March 2024 did not make it on-chain. A datum was published manually at 18:14 UTC after the anomaly was spotted through network monitoring.
Status
Under Review
Assessment
The validator node caused the websocket connection to the COOP publishing server to be dropped. The reliability of the websocket connection and potential changes to the publishing mechanism remain under investigation.
Additional Notes
While not directly related to INCIDENT 019 we know that the reliability of the websocket connection between the validator and publishing node can impact publication. How a heartbeat is requested and subsequent monitoring of its successful publication continues to be looked at as the Orcfax network is incrementally upgraded.
We are investigating:
Documentation improvements