Indexer Updater doesn't add necessary line to metastate table

[PSjoe]

Subject of the issue

An instance of Indexer configured to update the PostgreSQL database does not add the necessary "feesink" line to the metastate table until a call is made to /v2/accounts. Until this happens, any indexer instances that are configured with read-only access to the DB are unable to service this request. They return the error:

{"message":"failed while searching for account: problem saving metastate: pq: permission denied for table metastate"}

Your environment

We are running Indexer v2.6.1 and PostgreSQL 12.3 (Amazon hosted Aurora RDS)

Steps to reproduce

1. Start with an empty PostgreSQL DB
2. Configure two separate indexer instances, one that is updating the DB and a separate copy that has read-only access to the same DB
3. Attempt to query /v2/accounts from the read-only copy. The error above will be returned.
4. Attempt the same query on the updater and the query succeeds.
5. Attempt the same query again on the read-only indexer. The query now succeeds.

Expected behaviour

The read-only indexer should be able to serve all requests. The updater should insert all required data into the DB tables without requiring an API request.

Further details

While inspecting the DB "metastate" table, we can see what the table looks like before we make any queries against the updater:

mainnet=> select * from metastate;
     k     |                v                 
-----------+----------------------------------
 migration | {"next": 16}
 state     | {"next_account_round": 15735937}
(2 rows)

After making a query to the updater, the table looks like this:

mainnet=> select * from metastate;
     k     |                                                             v                                                              
-----------+----------------------------------------------------------------------------------------------------------------------------
 migration | {"next": 16}
 state     | {"next_account_round": 15736093}
 accounts  | {"FeeSink": "x/zNsljw1BicK/i21o7ml1CGQrCtAB8x/LkYw1S6hZo=", "RewardsPool": "/v////////////////////////////////////////8="}
(3 rows)

That third line is added to the table only after the /v2/accounts call is made to a copy of indexer that has write permissions. In our setup, we normally do not pass any REST calls to the indexer configured with write access.

This affects all three ledgers. I've confirmed that the PostgreSQL read-only user does have full read access to all tables in the DB. The above output from psql was created while using the read-only DB user.

I normally stand up DBs using a pre-existing snapshot and so would not run into this. I re-indexed all three ledgers from block 0 using indexer 2.6.1 and ran into this issue. The last time I re-indexed an entire DB was with Indexer 2.3.2. I do not recall running into this issue then. So this bug likely was introduced between 2.3.2 and 2.6.1

[PSjoe]

I should also mention that is happened against a DB that was fully caught up to the end of the chain.

[yaovi-a]

There's a fix on the way.

[winder]

This has been resolved.