search

hashgraph / guardian

The Guardian is an innovative open-source platform that streamlines the creation, management, and verification of digital environmental assets. It leverages a customizable Policy Workflow Engine and Web3 technology to ensure transparent and fraud-proof operations, making it a key tool for transforming sustainability practices and carbon markets.
Apache License 2.0
104 stars 131 forks source link

Message sequencing and traceability in failure/retry situations #1290

Open anvabr opened 2 years ago

anvabr commented 2 years ago

Problem description

When Guardian is producing hedera transactions sometimes they need to be chunked and submitted independently. If one of the chunk submission fails for some reason the system seem to be losing track of sequencing. For example

but

If this indicates that something has failed in IPFS accepting the 3 of 3 for 23785, as there is no other implicit way of defining the chain here which this breaks traceability.

Step to reproduce

See above

Expected behavior

Ability trace the sequence of messages - when they retried and succeeded it is possible to link them together

Screenshots

N/A - see description.

anvabr commented 2 years ago

@Stepan-Kirjakov has found all the messages, they have been sent as follows:

'{"id":"916bcc18-8b8d-4d2d-aa34-daed4401130a","status":"ISSUE","type":"VP-Document","action":"create-vp-document","lang":"en-US","issuer":null,"relationships":["1662344893.432885274","1662344903.878245003","1662344907.218756377","1662344916.912400003","1662344922.081336003","1662344925.967598219","1662344931.210037003","1662344939.397006003","1662344941.812727003","1662344947.943785384","1662344952.447012808","1662344956.283986003","1662344961.005475738","1662344964.376551137","1662344967.368988003","1662344970.399941003","1662344977.188053003","1662344984.246511600","1662344999.985191003","1662345002.519201003","1662345020.031007939","1662345039.566140255","1662345058.180487003","1662345080.708139258","1662345099.822536659","1662345120.135477847","1662345138.897236926","1662345159.018879431","1662345178.044535003","1662345200.330111717","1662345218.733757825","1662345239.858338633","1662345259.444371003","1662345279.501271003","1662345298.685728407","1662345317.861107003","1662345338.535223003","1662345360.76'

'5245003","1662345378.948506762","1662345398.235328003","1662345419.496059003","1662345439.692187003","1662345461.695603649","1662345479.530455845","1662345580.249586003","1662345599.565260003","1662345618.654592003","1662345639.462502865","1662345842.212895010","1662345847.218841003","1662345859.914185862","1662345865.429540003","1662345879.661745970","1662345883.452095019","1662345899.869290003","1662345904.428929386","1662345919.573554822","1662345923.101373003","1662345942.010821003","1662345944.559376745","1662345960.727179346","1662345963.497311765","1662345982.564893403","1662345986.034435836","1662345990.409187719","1662345994.960460353","1662345999.763680853","1662346004.769611826","1662346008.656791003","1662346013.213255003","1662346017.703008670","1662346028.243071632","1662346032.815575316","1662346038.773315003","1662346047.639850766","1662346052.690358358","1662346068.162766003","1662346074.564523003","1662346090.006147003","1662346094.071642600","1662346109.248225646","1662346113.453851988","16'

'62346128.520859003","1662346134.431349003","1662346138.644078003","1662346155.155349003"],"cid":"bafybeieavkgy54ac7wxqjapxuyrf5vqjsnvclkwwnnx2cecbwqvwth4esq","url":"https://ipfs.io/ipfs/bafybeieavkgy54ac7wxqjapxuyrf5vqjsnvclkwwnnx2cecbwqvwth4esq"}'

The message that was sent into Hedera topic was too large so the SDK broke it into 3 chunks, this has nothing to do with IPFS.

anvabr commented 2 years ago

Furthermore, as per @Stepan-Kirjakov all these messages have a common ID in: "transaction_valid_start":"1662346164.119901013"}, and therefore can be traced/joined together. There is no guarantees from Hedera SDK and no expectation that the messages are consecutive.

voycey commented 2 years ago

@anvabr thanks for this - I think it needs to be clear that transaction_valid_start is the ID that we should use for this as it was previously discussed that the sequence number was the way to identify this.

Also - I believe that there is no way to directly search for this? So this means we essentially have to iterate through each sequence number until we find a matching transaction_valid_start? This is horribly inefficient.

petreze commented 1 year ago

Hello @anvabr :wave:, from the provided links of the 3 messages I can see that the payer_account_id of the third message differs from the other two messages. So I would assume that the message is signed and executed by other client and persisted as a separate message. Could you please provide to us an example where you encounter this issue? Thanks!

anvabr commented 1 year ago

Note - created https://github.com/hashgraph/hedera-sdk-js/issues/1333, discussion on-going.

voycey commented 1 year ago

Hey guys - seeing as this is back under discussion.

I think there should be some "final" message / transaction that is pushed that is a map of previous transaction_valid_start or sequence numbers - this would mean in a single transaction we could retrieve the exact locations of the chunks - this would increase our efficiency significantly!

anvabr commented 1 year ago

Hey guys - seeing as this is back under discussion.

I think there should be some "final" message / transaction that is pushed that is a map of previous transaction_valid_start or sequence numbers - this would mean in a single transaction we could retrieve the exact locations of the chunks - this would increase our efficiency significantly!

@voycey I've started a discussion with hedera sdk team https://github.com/hashgraph/hedera-sdk-js/issues/1333, I feel it'd add more clout to that discussion if you also contribute/comment there please.