The datum at 18:39 & 1900 UTC were not published. A decoding error occurred within the COOP environment.
Status
Under Review
Assessment
The COOP environment failed to store two consecutive fact statements and generated the message, error submitting Tx, unable to store fact; the publications implicated:
A 1% deviation at 18:39 UTC
The hourly heartbeat at 19:00 UTC with the error
The issue looks to be related to Incident 15. The error occurred in the local fact store which is used as a temporary store during publishing. When the storage of the fact failed, the resulting transaction could not be built and published as anticipated.
The development team is working to identify the root cause of the error.
Heartbeat monitoring and publishing continued without issue after the missed heartbeat.
Additional Notes
For consumers, an Orcfax Fact Statement within the validity range was unavailable on-chain from 18:19 UTC until 20:00 UTC (approx. 1hr 40 mins).
The JSON format used by Orcfax remains stable and the fact statement has been tested in a preprod environment without issue, as such the root cause that affected mainnet services remains under review.
Technical improvements
We are investigating:
Continuing to find more effective ways of debugging the COOP environment so as to gather as much information as possible.
Publishing and transaction building components of COOP are currently being rewritten and issues such as these will be incorporated in further testing.
Continuing to investigate mechanisms for resilient workflows in the Orcfax environment to ensure reliable publishing.
Trigger
Date
2024-01-23
Summary
The datum at 18:39 & 1900 UTC were not published. A decoding error occurred within the COOP environment.
Status
Under Review
Assessment
The COOP environment failed to store two consecutive fact statements and generated the message,
error submitting Tx, unable to store fact; the publications implicated:The issue looks to be related to Incident 15. The error occurred in the local fact store which is used as a temporary store during publishing. When the storage of the fact failed, the resulting transaction could not be built and published as anticipated.
The development team is working to identify the root cause of the error. Heartbeat monitoring and publishing continued without issue after the missed heartbeat.
Additional Notes
For consumers, an Orcfax Fact Statement within the validity range was unavailable on-chain from 18:19 UTC until 20:00 UTC (approx. 1hr 40 mins).
The JSON format used by Orcfax remains stable and the fact statement has been tested in a preprod environment without issue, as such the root cause that affected mainnet services remains under review.
Technical improvements
We are investigating:
Documentation improvements
NA