Add custom indexer solution for fetching migration shared objects by expiration unlock condition address fields

[begonaalvarezd]

It seems not possible to fetch stardust shared objects by the subfields owner and return_address of expiration_unlock_condition. Since these are internal fields and only encoded in move, they seem not indexed. We tried using GraphQL and couldnt make it work, though this is not 100% confirmed

There are 2 options so we (and anybody) can fetch these objects:

1. Write a small app based on the custom indexing framework which indexes only stardust stuff specifically and exposes it via some rudimentary API
2. Add extra indices specifically for this in the graphql api

Since this is a one-time thing, the winner solution seems to be option 1.

Description

Expose a simple REST API that can be used to fetch shared objects by address and return all stardust shared objects that contain an expiration unlock condition and that address is either the owner or the return_address. The response should be paginated and contain all the fields returned by a simple owned object query, eg

{
  "id": "0x1fc6eb2c566b0fdb2c19f05ebb2c5311c523628c77102f0f0354a8cd0f8cd156",
  "balance": {
    "value": "1000000000"
  },
  "native_tokens": {
    "id": "0x614d543fe15e768b30dbc40c7731c404ce94039e20f6aba556b2cd68f760943b",
    "size": "0"
  },
  "storage_deposit_return_uc": null,
  "timelock_uc": null,
  "expiration_uc": {
    "owner": "0x50a53acf9e8d19107b2420b588ed747da79b26195539225f311b4f9446f41860",
    "return_address": "0x5a3bebb2e94873a904daaf4362e608034715f5dadb03d7b7571f5e2b06bcf161",
    "unix_time": 1730367277
  },
  "metadata": [],
  "tag": [],
  "sender": "0x50a53acf9e8d19107b2420b588ed747da79b26195539225f311b4f9446f41860"
}

The logic to filter by who the owner is at a given time/epoch can be done by the different apps using this api.

---

Extra

• Ideally, for testing purposes, we should be able to monitor custom stardust packages deployed in dev networks, in tooling we have our own stardust packages that we use to mint migration testing scenarios
• This custom indexer should be available in all testing networks, including localnet

[kodemartin]

On querying basic outputs with the current solutions

The contents of Move objects are stored in the database as a data blob, and this implies that we cannot query objects based on their contents.

So it is indeed impossible to query basic outputs by the expiration unlock condition (EUC) fields.

It is possible nevertheless to query objects based on their type in GraphQL. So with a query of the sort:

{
  objects(filter: {type:"0x107a::basic_output::BasicOutput<0x2::iota::IOTA>"} ) {
    nodes {
      address
      owner {
        __typename
      }
      asMoveObject {
        contents {
          json
        }
      }
    }
  }
}

we can get paginated results on all basic outputs, and their contents after they are properly deserialized. This is of course far from ideal.

The ideal solution would be to somehow provide a relation (e.g. an in-memory map, or a database table) between the ids of the basic outputs and the fields of the EUCs, providing reverse indexes on the EUC owner and return_address fields. The default indexer database could be then queried based on the object id and resolve the object.

Possible solutions on indexing EUCs

In order to create an EUC relation of this sort, it seems that it would suffice to process only the genesis checkpoint. That is, we don't necessarily have to implement a custom indexer based on iota-data-ingestion framework to process all checkpoints.

The application serving the EUC relation data through a REST API could use any of the following patterns:

1. Spawn a GraphQL client to query basic outputs and construct on load the EUC relation either in-memory, or on disk (through SQLite for example).

   Pros: This would be the simplest approach in terms of setup, only requiring a GraphQL server running remotely. Cons: It depends on the synchronization status of the default indexer database supporting the GraphQL server. It would also require custom logic to process the results to create the EUC relation.

2. Spawn an iota-rest-api client and fetch only the genesis checkpoint to construct on load the EUC relation.

   Pros: This fetches directly the genesis checkpoint from the node, and can use existing logic in processing the . Cons: This would require a collocated full-node upon deployment.

3. Spawn a custom indexer using the data-ingestion framework, syncing with a full-node through its Rest API.

   Pros: This requires minimal additional logic, and can be extended in the future to satisfy additional custom indexing use cases. Cons: The data-ingestion framework spawns a worker that processes every new checkpoint, which is an overkill for the problem at hand.

On monitoring custom stardust packages

I interpret this part of the feature description

Ideally, for testing purposes, we should be able to monitor custom stardust packages deployed in dev networks, in tooling we have our own stardust packages that we use to mint migration testing scenarios

as a request to be able to query a package that depends on the stardust package, which resolves into being able to query a package based on its transitive dependencies. This is not supported by the default indexer solution, as the package contents are also stored in a serialized form.

To accommodate such a request, a custom indexer (so option (3)) above seems to be the way to go.

cc: @begonaalvarezd @msarcev @lzpap

[brancoder]

For the first two options, after a user performs a migration transaction, how would the EUC relation be removed from the DB for the migrated objects?

[kodemartin]

For the first two options, after a user performs a migration transaction, how would the EUC relation be removed from the DB for the migrated objects?

The idea is not to update the EUC relation at all. Once we get the object id, the indexer database will tell us if the object has been deleted or is still alive.