[Bug] Reassigning subgraph to non-existent indexer doesn't stop indexing

[paymog]

Bug report

It seems that reassigning a subgraph to a non-existent indexer does not stop indexing on 0.33.0. I have a subgraph that's erroring out and I want to reassign it to an indexer that doesn't exist to stop indexing. I know there's a new pause command but that doesn't have a JSON HTTP API so I can't build automation around it.

I just tried reassigning with the following command on the command line. I see that the removal event was received by the graph node indexer, but the indexers continue to index

 ❯❯❯ kubectl exec --namespace default -it deploy/unicrypt-indexer-0-graph-node -- graphman reassign QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL paused2
reassigning QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL[1264] to paused2 (was paused)
warning: this is the only deployment assigned to paused2
         are you sure it is spelled correctly?

Relevant log output

unicrypt-indexer-0-graph-node-75ff4dbbff-xr9cc graph-node Nov 22 10:06:02.990 DEBG Received assignment event: Remove { deployment: DeploymentLocator { id: DeploymentId(1264), hash: DeploymentHash("QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL") }, node_id: NodeId("indexer_0") }, component: SubgraphRegistrar
unicrypt-indexer-1-graph-node-7dd9dbc6c8-bw4wr graph-node Nov 22 10:06:02.991 DEBG Received assignment event: Remove { deployment: DeploymentLocator { id: DeploymentId(1264), hash: DeploymentHash("QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL") }, node_id: NodeId("indexer_1") }, component: SubgraphRegistrar
unicrypt-indexer-0-graph-node-75ff4dbbff-xr9cc graph-node Nov 22 10:06:23.540 DEBG Starting or restarting subgraph, sgd: 1264, subgraph_id: QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL, component: SubgraphInstanceManager
unicrypt-indexer-0-graph-node-75ff4dbbff-xr9cc graph-node Nov 22 10:06:23.540 DEBG Starting block stream, sgd: 1264, subgraph_id: QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL, component: SubgraphInstanceManager
unicrypt-indexer-0-graph-node-75ff4dbbff-xr9cc graph-node Nov 22 10:06:23.540 INFO Blockstream disconnected, connecting, provider_err_count: 0, cursor: TtiCG_7MSiQkWj2ONQIgHKWwLpc_DFttUAriIhBJj4-j9yDGjJqhBmB3YRXTk6j3iRK5Qwn639_LHSsopsYBuYC5l-xlvyBrEnN-k4G4_r3oKaGkawhJculkWuLfMN7RWj7SYw3ze7kK5IDgaPCIYEU0Y8UvJGWwjjgEpddcI_UQuXU8lm-pdZ7XgajCpYMT_rFxTLahnX2mDjIsfhkIP56BNfSat21yYw==, subgraph: QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL, start_block: 12517819, endpoint_uri: streamingfast-firehose-bsc, provider: streamingfast-firehose-bsc, deployment: QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL, sgd: 1264, subgraph_id: QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL, component: FirehoseBlockStream

IPFS hash

QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL

Subgraph name or link to explorer

No response

Some information to help us out

• [X] Tick this box if this bug is caused by a regression found in the latest release.
• [ ] Tick this box if this bug is specific to the hosted service.
• [X] I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux

[paymog]

It seems that even using graphman pause QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL does not stop the subgraph from indexing

[paymog]

turns out restarting both indexers in my cluster forced the subgraph to stop indexing. Something about the messaging between graph nodes to pause indexing was broken.

[fordN]

Closing as it seemed to be a one-off issue.