Farm pricing policy not in sync

Update: The indexer should not be blamed for this issue.

The farms were initially created with a hard-coded pricing policy 1, but the old implementation of update_farm extrinsic allowed farmers to change the pricing ID based on their parameters.

The old implementation is archived here: https://github.com/threefoldtecharchive/tfchain_pallets/blob/c60225c88cf94f68fc93202a929bd2b721b8f977/pallet-tfgrid/src/lib.rs#L366C60-L366C77

This was an issue, and a ticket was opened here to fix it: https://github.com/threefoldtech/tfchain/issues/501

however, Before the fix being implemented and deployed, some farmers updated their farm's name but left the pricing policy field with a default value of 0. Thus, the indexer stack indexed these farms as expected, based on the incoming on-chain events, with pricing policy ID 0.

Here are the FarmUpdated events from updatd_farm extrinsic related to these farms where the pricing policy changed to 0

{
  "data": {
    "events": [
      {
        "name": "TfgridModule.FarmUpdated",
        "args": {
          "certificationType": {
            "__kind": "Diy"
          },
          "dedicatedFarm": false,
          "id": 57,
          "name": "0x42696c626f",
          "pricingPolicyId": 0,
          "publicIps": [],
          "twinId": 247,
          "version": 2
        },
        "id": "0002207831-000001-e30f9",
        "block": {
          "height": 2207831,
          "timestamp": "2022-05-31T22:46:48.000000Z",
          "validator": "0xae68a346b12562eabbbb4f90e7654522f4608a09eac09a74c2ebc57aaa32c70e",
          "id": "0002207831-e30f9",
          "spec": {
            "specVersion": 56,
            "blockHeight": 1944966
          }
        }
      }
    ]
  }
}

{
  "data": {
    "events": [
      {
        "name": "TfgridModule.FarmUpdated",
        "args": {
          "certificationType": {
            "__kind": "Diy"
          },
          "id": 355,
          "name": "0x456e6b796661726d4f6c64",
          "pricingPolicyId": 0,
          "publicIps": [],
          "twinId": 1076,
          "version": 1
        },
        "id": "0001424082-000001-2015e",
        "block": {
          "height": 1424082,
          "timestamp": "2022-04-07T11:07:48.000000Z",
          "validator": "0x7478f032b9d3fdd3c9f29880b7b28d6de9ec2e106b92b41f0ea71e0d9f3f1f19",
          "id": "0001424082-2015e",
          "spec": {
            "specVersion": 49,
            "blockHeight": 1307667
          }
        }
      }
    ]
  }
}

{
  "data": {
    "events": [
      {
        "name": "TfgridModule.FarmUpdated",
        "args": {
          "certificationType": {
            "__kind": "Diy"
          },
          "dedicatedFarm": false,
          "id": 2097,
          "name": "0x5468654461726b546f776572",
          "pricingPolicyId": 0,
          "publicIps": [],
          "twinId": 5286,
          "version": 2
        },
        "id": "0002582978-000001-c1ebe",
        "block": {
          "height": 2582978,
          "timestamp": "2022-06-27T00:08:12.000000Z",
          "validator": "0x7478f032b9d3fdd3c9f29880b7b28d6de9ec2e106b92b41f0ea71e0d9f3f1f19",
          "id": "0002582978-c1ebe",
          "spec": {
            "specVersion": 56,
            "blockHeight": 1944966
          }
        }
      }
    ]
  }
}

{
  "data": {
    "events": [
      {
        "name": "TfgridModule.FarmUpdated",
        "args": {
          "certificationType": {
            "__kind": "Diy"
          },
          "dedicatedFarm": false,
          "id": 2162,
          "name": "0x73636f74742d7465737432",
          "pricingPolicyId": 0,
          "publicIps": [],
          "twinId": 414,
          "version": 2
        },
        "id": "0002207264-000001-92182",
        "block": {
          "height": 2207264,
          "timestamp": "2022-05-31T21:50:06.000000Z",
          "validator": "0x4ae560eaa83546dc54c33123d5602dcbfaf3edf9a8cabd446df54768a7e5d21b",
          "id": "0002207264-92182",
          "spec": {
            "specVersion": 56,
            "blockHeight": 1944966
          }
        }
      }
    ]
  }
}

{
  "data": {
    "events": [
      {
        "name": "TfgridModule.FarmUpdated",
        "args": {
          "certificationType": {
            "__kind": "Diy"
          },
          "dedicatedFarm": false,
          "id": 2325,
          "name": "0x4e412d50524f442d31",
          "pricingPolicyId": 0,
          "publicIps": [],
          "twinId": 5924,
          "version": 2
        },
        "id": "0002721520-000001-19084",
        "block": {
          "height": 2721520,
          "timestamp": "2022-07-06T15:02:36.000000Z",
          "validator": "0x7478f032b9d3fdd3c9f29880b7b28d6de9ec2e106b92b41f0ea71e0d9f3f1f19",
          "id": "0002721520-19084",
          "spec": {
            "specVersion": 56,
            "blockHeight": 1944966
          }
        }
      }
    ]
  }
}

Later, the storage of these farms was reset back to pricing policy ID 1 via a storage migration that was performed to fix it for the farmers who unintentionally changed this value. This storage migration has set the pricing policy ID to 1 for all farms without taking into consideration emitting necessary events, resulting in inconsistent data.

https://github.com/threefoldtech/tfchain/blob/eb36aa90df2d60cb1a534997903821fc68a096f1/substrate-node/pallets/pallet-tfgrid/src/migrations/v11.rs#L76

The good is that update_farm can no longer change the pricing policy, so the issue won't happen again.

However, there have been recurring issues that are similar to this one. These issues are all caused by old migrations that alter chain storage without emitting the expected events. As a result, our indexer has not been notified of such changes, and the data has become inconsistent. Fixing these issues is not a straightforward process and can be ugly or quite complicated to maintain.

threefoldtech / tfchain_graphql

Farm pricing policy not in sync #96