Fix error related to indexer running against redwood

[wgwz]

In staging indexer running against redwood we're seeing this error:

2023-08-22T18:08:35.113407+00:00 app[index-regen.1]: 2023-08-22 18:08:35,113 - utils - ERROR - target=<function _index_class_issuers at 0x7f71a3f1ae60> pid=10 exc=500 Server Error: Internal Server Error for url: http://redwood.regen.network:1317/regen/ecocredit/v1/classes/C60/issuers
2023-08-22T18:08:35.113407+00:00 app[index-regen.1]: Traceback (most recent call last):
2023-08-22T18:08:35.113407+00:00 app[index-regen.1]: File "/app/utils.py", line 87, in run
2023-08-22T18:08:35.113407+00:00 app[index-regen.1]: self._target(self.pg_conn, self.client, chain_num)
2023-08-22T18:08:35.113408+00:00 app[index-regen.1]: File "/app/index_class_issuers.py", line 49, in _index_class_issuers
2023-08-22T18:08:35.113408+00:00 app[index-regen.1]: issuers = fetch_class_issuers(
2023-08-22T18:08:35.113408+00:00 app[index-regen.1]: File "/app/index_class_issuers.py", line 15, in fetch_class_issuers
2023-08-22T18:08:35.113408+00:00 app[index-regen.1]: resp.raise_for_status()
2023-08-22T18:08:35.113408+00:00 app[index-regen.1]: File "/app/.heroku/python/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
2023-08-22T18:08:35.113409+00:00 app[index-regen.1]: raise HTTPError(http_error_msg, response=self)
2023-08-22T18:08:35.113409+00:00 app[index-regen.1]: requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://redwood.regen.network:1317/regen/ecocredit/v1/classes/C60/issuers
2023-08-22T18:08:35.114820+00:00 app[index-regen.1]: 2023-08-22 18:08:35,114 - index_class_issuers - INFO - {'type': 'regen.ecocredit.v1.EventCreateClass', 'block_height': 9820936, 'tx_idx': 2, 'msg_idx': 0, 'chain_num': 1, 'timestamp': '2023-07-05T12:26:24Z', 'tx_hash': '1dc0d47b5a0b211ea42479973ce541f33e813a639df7c7f145087033224184dc', 'class_id': 'C60'}

This is to do with the fact that we issue this REST api call with x-cosmos-block-height header set to the block height of the event.

Querying the issuers w/o x-cosmos-block-height returns 200:

➜  analytics git:(main) ✗ http GET http://redwood.regen.network:1317/regen/ecocredit/v1/classes/C60/issuers
HTTP/1.1 200 OK
Content-Length: 135
Content-Type: application/json
Date: Tue, 22 Aug 2023 18:12:50 GMT
Grpc-Metadata-Content-Type: application/grpc+cosmos-sdk-grpc-codec
Grpc-Metadata-X-Cosmos-Block-Height: 10557993
X-Server-Time: 1692727970

{
    "issuers": [
        "regen1df675r9vnf7pdedn4sf26svdsem3ugavgxmy46"
    ],
    "pagination": {
        "next_key": null,
        "total": "1"
    }
}

Querying w/ x-cosmos-block-height set returns the 500 error:

➜  analytics git:(main) ✗ http GET http://redwood.regen.network:1317/regen/ecocredit/v1/classes/C60/issuers x-cosmos-block-height:9820936
HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Date: Tue, 22 Aug 2023 18:13:31 GMT
Trailer: Grpc-Trailer-Content-Type
Transfer-Encoding: chunked
X-Server-Time: 1692728011

{
    "code": 2,
    "details": [],
    "message": "codespace sdk code 18: invalid request: failed to load state at height 9820936; version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height (latest height: 10558000)"
}

I think we don't see this issue in production since we're using the archive node (since this is deployed to production and running fine there).

[wgwz]

As a temporary workaround I've deployed the following change to the staging indexer:

diff --git a/index_class_issuers.py b/index_class_issuers.py
index 2c635cd..327cf90 100644
--- a/index_class_issuers.py
+++ b/index_class_issuers.py
@@ -10,7 +10,7 @@ logger = logging.getLogger(__name__)
 def fetch_class_issuers(height, class_id):
     resp = requests.get(
         f"{os.environ['REGEN_API']}/regen/ecocredit/v1/classes/{class_id}/issuers",
-        headers={"x-cosmos-block-height": str(height)},
+        # headers={"x-cosmos-block-height": str(height)},
     )
     resp.raise_for_status()
     return resp.json()["issuers"]

[ryanchristo]

Maybe we should have a configuration variable to determine whether using an archive node or not.

[ryanchristo]

Hey team! Please add your planning poker estimate with Zenhub @ryanchristo @wgwz @blushi