Gateway discards information about a transaction's index within block and bundle

Hello,

I am designing a dApp where the order of confirmed transactions on the Arweave network affects the application state. However, it appears that the arweave protocol does not define a canonical algorithm for transaction order within blocks.

The most sensible algorithm in my opinion is:

order by BLOCK, then
order by LAYER 1 TRANSACTION in the order they appear in the block, then
if the layer 1 transaction is a bundle, the order in which they appear in the bundle

However, information about the index of layer 1 transactions within the block, as well as information about the order of a bundled transaction within its bundle is DISCARDED by the gateway, and not accessible via the GraphQL API. I propose to instead keep this data and make it available on the GraphQL transaction model.

Additionally, the sort order of transaction queries might be modified to sort transactions using the algorithm stated above.

Relevant conversation on the Arweave Dev Talk #dev channel

[10:09 AM]  myn: The reason I'm concerned about this is that I'm designing a dApp where the order that the transactions appear on chain determines the application state.
[10:09 AM]  DMac: pretty sure vartex doesn't work 100% yet
[10:09 AM]  myn: Oh right, what powers arweave.net/graphql then?
[10:11 AM]  DMac: https://github.com/ar-io/arweave-gateway
...
[10:15 AM]  Ros: pretty sure there is no ordering at the protocol level?
[10:15 AM]  Ros: random txs get pulled from the mempool to create a new block if i'm not mistaken
[10:16 AM]  DMac: redstone SWC gateway enforces a deterministic order
[10:16 AM]  myn: But I'm talking about once the block is mined, surely those transactions are ordered
[10:16 AM]  DMac: but that's not for data transactions
[10:16 AM]  DMac: just sort them by txid
[10:17 AM]  myn: This works I guess
[10:17 AM]  Ros: not sure why the miner needs to spend extra time ordering txids
[10:18 AM]  myn: I'm not saying they need to put them in a particular order, I'm asking about whether the order is persisted (and consistent) 
[10:18 AM]  DMac: you're more asking weather GQL will return them in the same order
[10:19 AM]  myn: Well, both
[10:19 AM]  DMac: you can grab the block, unbundle any tX bundles
[10:19 AM]  DMac: and list them in order yourself
[10:19 AM]  myn: Got it
[10:19 AM]  DMac: that order won't change
[10:19 AM]  DMac: but if you query graphQL.. ideally the same order would exist
[10:19 AM]  myn: So essentially the information is there in the block, but the graphQL endpoints lose that information when indexing
[10:19 AM]  DMac: but I don't know that there are any guarantees
[10:20 AM]  DMac: possibly,  the block ingesting could be parallelized
[10:20 AM]  DMac: so the order isn't 100%
[10:20 AM]  myn: Understood, thanks a bunch @DMac
[10:20 AM]  Ros: check out this example block header https://arweave.net/block/height/964000
[10:21 AM]  Ros: it's a json file with the included txs in a txid array. they stay in that order. @myn 
[10:22 AM]  myn: Perfect, this answers my question
[10:22 AM]  DMac: challenge here is bundles Ros
[10:22 AM]  DMac: do the bundled TX get injected into that order? or appended?
[10:23 AM]  Ros: there is a txid index.
[10:23 AM]  Ros: well dataItem ids really
[10:23 AM]  DMac: right.. but say there's 5 bundle tx in a block
[10:23 AM]  DMac: say their tx indexes are 1-5
[10:23 AM]  DMac: the gateway fires up a job, one for each unbundle
[10:24 AM]  DMac: do the bundled tx get appended to the indexes in the block in the order they complete
[10:24 AM]  DMac: or the order the base TX appear in the block?
[10:24 AM]  DMac: some of this behaviour is a little undefined
[10:24 AM]  Ros: you could also have the same txs in the same or multiple blocks in different bundles 😬 🤯
[10:25 AM]  Ros: which one is the "real" dataItem?
[10:25 AM]  DMac: so the order of L1 tx are prety deterministic
[10:25 AM]  DMac: but bundled tx are a bit.. .. uhh...
[10:25 AM]  DMac: who knows
[10:27 AM]  myn: I guess if the same transaction can exist multiple times then it makes sense that the first one is the canonical one
[10:28 AM]  myn: I guess my point is that it would be nice if there was a standardized way to order transactions from first to last... rather than leave it as an implementation detail for any protocol developers 😅
[10:28 AM]  DMac: sort by txid per block 😄
[10:28 AM]  DMac: nothing else is deterministic really
[10:29 AM]  myn: Yes that's possible, but again it's an arbitrary implementation detail (and in my opinion a workaround for the GQL gateways losing information about transaction order within the block) 
[10:29 AM]  DMac: like.. you can try and enforce a unix-timestamp tag on the TX
[10:29 AM]  DMac: but people can upload with any tags they like
[10:29 AM]  DMac: it's not really reliable
[10:29 AM]  Ros: the unbundler defo makes a decision regarding which block a duplicate dataItem belongs to.
confusingly graphql used to use the last ans102 dataItem uploaded
[10:29 AM]  Ros: not sure what is standard now
[10:30 AM]  myn: I'm not suggesting this
[10:30 AM]  DMac: oh yeah, was just a hypothetical suggestion by me
[10:32 AM]  myn: I think the real solution here is that GraphQL transaction entites include a block index field, and an optional bundle index field, and the protocol defines a canonical order comparitor function based on these
[10:33 AM]  myn: I'll open an issue on https://github.com/ar-io/arweave-gateway
[10:33 AM]  DMac: that would be a cool property to add to gql
[10:33 AM]  DMac: the maintainers of the gateway are over at https://discord.gg/Hk8vFKpJ
[10:33 AM]  DMac: if you wanna suggest and chat there
[10:33 AM]  myn: Understood, thanks @DMac ❤️

ar-io / arweave-gateway

Gateway discards information about a transaction's index within block and bundle #1