`forc index deploy --replace-indexer` not working as expected

[ra0x3]

• Please see this Slack thread for context
  • We confirmed that the error happened on http://beta-4.fuel.network as well
  • This means that --replace-indexer is not actually replacing the indexer with a new indexer. How do I know?
    • First I removed the old indexer with forc index remove
    • Then I redeployed the explorer starting at block 7,000,000 and it works - no deserialization error
• This all leads me to believe that for whatever reason - --replace-indexer is not actually replacing the indexer properly
• I was not able to reproduce this locally - the explorer worked from block 7,000,000 with no issue
  • However, locally I did not try to use --replace-indexer
  • If you really wanted to repro you could:
  • Deploy an indexer using the last v0.23 release we had -- locally, on beta-4, starting from block 7,000,000
  • Let it index a few thousand blocks
  • Redeploy that indexer with v0.24.1 (without stopping the other one) and using --replace-data)
• Expected results
  • I would expect that the old indexer bytes get removed, and the new indexer bytes get registered - and the indexer executor task gets re-created, and indexing continues smoothly
  • The fact that I had to forc index remove before forc index deploy would work tells me the old indexer bytes must not have been removed 🤔

[ra0x3]

cc @lostman

[lostman]

@ra0x3, did you deploy with --replace-indexer and --remove-data?

If you did forc index remove, it would remove the indexer, and it's data.

However, forc index deploy --replace-indexer would only remove the indexer, not the data. So, if something and the serialized data stored in the database were no longer compatible, we'd get the deserialization error. I think the error was coming from get_object, so it would make sense.

We have a check that the deployed indexer has to be built with tooling with the same version. However, I don't think this applies to an already deployed indexer. Perhaps we could at least print something like:

WARN Indexer(some_indexer) was compiled with toolchain version A.B.C while the indexer service version is A.B.C. This may cause errors.

[ra0x3]

@lostman

• I did not use --remove-data because I did not want to remove the data
• I was forced to use forc index remove because forc index deploy --replace-indexer was simply (for whatever reason) not working
• I do not believe the QA env is running the indexer version check

  Containers:
  fuel-indexer:
  Container ID:  containerd://1a3be8b56d343c06090867a99cb391026c94836cabfc2693f1f5f83777e22c8d
  Image:         ghcr.io/fuellabs/fuel-indexer:sha-118b6bd
  Image ID:      ghcr.io/fuellabs/fuel-indexer@sha256:9d1fbc97ea109c345db69fa91a6d855483bfec63cfba157ed6047c4735a1b118
  Port:          29987/TCP
  Host Port:     0/TCP
  Command:
    ./fuel-indexer
  Args:
    run
    --fuel-node-host
    beta-4.fuel.network
    --fuel-node-port
    80
    --postgres-password
    wlxwRcpcxiLpJaNKjJafpeLvIzQHvB
    --postgres-host
    indexer-dev-001.c9ddshmlrizp.us-east-1.rds.amazonaws.com
    --postgres-port
    5432
    --postgres-user
    postgres
    --postgres-database
    postgres
    --web-api-host
    0.0.0.0
    --run-migrations
    --auth-enabled
    --auth-strategy
    jwt
    --jwt-secret
    UiKWKUCzrECfwCXnOSJdnOPeuUFHbXRyCeixCweYkptEFHyryJ
    --jwt-issuer
    FuelLabs
    --rate-limit
    --rate-limit-request-count
    3
    --rate-limit-window-size
    10
    --verbose
    --replace-indexer
    --accept-sql-queries
    --block-page-size

• We should update this in a PR

[ra0x3]

@lostman Also keep in mind that --disable-toolchain-version-check is disabled by default, so if a version mismatch was the issue we would've saw that error when I was deploying