indexer-agent: is trying to open allocations non stop on mips testnet

begetan commented 2 years ago

It looks like indexer agent is trying to open allocations despite of they are already processed onchain.

indexer version 0.20.4

Set deployment rules

graph indexer rules set QmWq1pmnhEvx25qxpYYj9Yp6E1xMKMVoUjXVQBxUJmreSe allocationAmount 10000 decisionBasis always parallelAllocations 1 autoRenewal true
graph indexer rules set QmSqJEGHp1PcgvBYKFF2u8vhJZt8JTq18EV7mCuuZZiutX allocationAmount 10000 decisionBasis always parallelAllocations 1 autoRenewal true
graph indexer rules set QmeVXKzGKSyfEQib4MzeZveJgDYJCYDMMHc1pPevWeSbsq allocationAmount 10000 decisionBasis always parallelAllocations 1 autoRenewal true

Indexer agent is trying to open the same allocations again and again:

$graph indexer actions get
✔ Processed input parameters
✔ Actions query returned
┌────┬──────────┬────────────────────────────────────────────────┬──────────────┬────────┬──────┬───────┬──────────┬─────────┬──────────────┬───────────────┬────────────────────────────────────────────────────────────────────┬───────────────────┐
│ id │ type     │ deploymentID                                   │ allocationID │ amount │ poi  │ force │ priority │ status  │ source       │ failureReason │ transaction                                                        │ reason            │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 12 │ allocate │ QmeVXKzGKSyfEQib4MzeZveJgDYJCYDMMHc1pPevWeSbsq │ null         │ 1000.0 │ null │ null  │ 0        │ failed  │ indexerAgent │ IE066         │ null                                                               │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 11 │ allocate │ QmSqJEGHp1PcgvBYKFF2u8vhJZt8JTq18EV7mCuuZZiutX │ null         │ 1000.0 │ null │ null  │ 0        │ failed  │ indexerAgent │ IE066         │ null                                                               │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 10 │ allocate │ QmW8Cbb2R4ZHWGsrYjNJKRjoKKcPeDTNK6rdipfQQaAhd6 │ null         │ 1000.0 │ null │ null  │ 0        │ failed  │ indexerAgent │ IE066         │ null                                                               │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 9  │ allocate │ QmWq1pmnhEvx25qxpYYj9Yp6E1xMKMVoUjXVQBxUJmreSe │ null         │ 1000.0 │ null │ null  │ 0        │ failed  │ indexerAgent │ IE066         │ null                                                               │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 8  │ allocate │ QmW8Cbb2R4ZHWGsrYjNJKRjoKKcPeDTNK6rdipfQQaAhd6 │ null         │ 1000.0 │ null │ null  │ 0        │ failed  │ indexerAgent │ IE066         │ null                                                               │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 7  │ allocate │ QmWq1pmnhEvx25qxpYYj9Yp6E1xMKMVoUjXVQBxUJmreSe │ null         │ 1000.0 │ null │ null  │ 0        │ failed  │ indexerAgent │ IE066         │ null                                                               │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 6  │ allocate │ QmSqJEGHp1PcgvBYKFF2u8vhJZt8JTq18EV7mCuuZZiutX │ null         │ 1000.0 │ null │ null  │ 0        │ failed  │ indexerAgent │ IE066         │ null                                                               │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 5  │ allocate │ QmeVXKzGKSyfEQib4MzeZveJgDYJCYDMMHc1pPevWeSbsq │ null         │ 1000.0 │ null │ null  │ 0        │ failed  │ indexerAgent │ IE066         │ null                                                               │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 4  │ allocate │ QmSqJEGHp1PcgvBYKFF2u8vhJZt8JTq18EV7mCuuZZiutX │ null         │ 1000.0 │ null │ null  │ 0        │ success │ indexerAgent │ null          │ 0x72fe2a6db2422ff709c0e3e2075ee206d656a8954264099dfac874f6e91e19bc │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 3  │ allocate │ QmeVXKzGKSyfEQib4MzeZveJgDYJCYDMMHc1pPevWeSbsq │ null         │ 1000.0 │ null │ null  │ 0        │ success │ indexerAgent │ null          │ 0x72fe2a6db2422ff709c0e3e2075ee206d656a8954264099dfac874f6e91e19bc │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 2  │ allocate │ QmW8Cbb2R4ZHWGsrYjNJKRjoKKcPeDTNK6rdipfQQaAhd6 │ null         │ 1000.0 │ null │ null  │ 0        │ success │ indexerAgent │ null          │ 0x72fe2a6db2422ff709c0e3e2075ee206d656a8954264099dfac874f6e91e19bc │ deployment:always │
├────┼──────────┼────────────────────────────────────────────────┼──────────────┼────────┼──────┼───────┼──────────┼─────────┼──────────────┼───────────────┼────────────────────────────────────────────────────────────────────┼───────────────────┤
│ 1  │ allocate │ QmWq1pmnhEvx25qxpYYj9Yp6E1xMKMVoUjXVQBxUJmreSe │ null         │ 1000.0 │ null │ null  │ 0        │ success │ indexerAgent │ null          │ 0x72fe2a6db2422ff709c0e3e2075ee206d656a8954264099dfac874f6e91e19bc │ deployment:always │
└────┴──────────┴────────────────────────────────────────────────┴──────────────┴────────┴──────┴───────┴──────────┴─────────┴──────────────┴───────────────┴────────────────────────────────────────────────────────────────────┴───────────────────┘```

Indexer agent logs:
```json
docker logs -f -n 1 indexer-agent |jq
{
  "level": 20,
  "time": 1665261378167,
  "pid": 1,
  "hostname": "edbc09430a5d",
  "name": "IndexerAgent",
  "results": [
    {
      "actionID": 13,
      "failureReason": "IE066"
    },
    {
      "actionID": 14,
      "failureReason": "IE066"
    },
    {
      "actionID": 15,
      "failureReason": "IE066"
    },
    {
      "actionID": 16,
      "failureReason": "IE066"
    }
  ],
  "msg": "Completed batch action execution"
}

How to reproduce:

Wipe indexer-agent database ( StakeSquid docker-compose)
Add cli rules for 4 testnet subgraphs
Check queue with cli: graph indexer actions get

After several hours it already open 16 allocations. It looks like every 2 hours it adds new allocations

papadritta commented 2 years ago

I have the same issue with subgraphs allocations. Build & install indexer-agent from source, version v0.20.3. Run with pm2 as a script. (node v14.20.0)

Set the rules for subgraphs:

graph indexer rules set QmW8Cbb2R4ZHWGsrYjNJKRjoKKcPeDTNK6rdipfQQaAhd6 parallelAllocations 1
graph indexer rules set QmWq1pmnhEvx25qxpYYj9Yp6E1xMKMVoUjXVQBxUJmreSe parallelAllocations 1
graph indexer rules set QmSqJEGHp1PcgvBYKFF2u8vhJZt8JTq18EV7mCuuZZiutX parallelAllocations 1
graph indexer rules set QmeVXKzGKSyfEQib4MzeZveJgDYJCYDMMHc1pPevWeSbsq parallelAllocations 1

$graph indexer rules get all

─────┬──────────┬─────────────────────┬────────┬───────────────┬──────────────────┐
│ identifier                                     │ identifierType │ allocationAmount │ allocationLifetime │ autoRenewal │ parallelAllocations │ maxAllocationPercentage │ minSignal │ maxSignal │ minStake │ minAverageQueryFees │ custom │ decisionBasis │ requireSupported │
├────────────────────────────────────────────────┼────────────────┼──────────────────┼────────────────────┼─────────────┼─────────────────────┼─────────────────────────┼───────────┼───────────┼──────────┼─────────────────────┼────────┼───────────────┼──────────────────┤
│ global                                         │ group          │ 100.0            │ null               │ true        │ 1                   │ null                    │ null      │ null      │ null     │ null                │ null   │ rules         │ true             │
├────────────────────────────────────────────────┼────────────────┼──────────────────┼────────────────────┼─────────────┼─────────────────────┼─────────────────────────┼───────────┼───────────┼──────────┼─────────────────────┼────────┼───────────────┼──────────────────┤
│ QmeVXKzGKSyfEQib4MzeZveJgDYJCYDMMHc1pPevWeSbsq │ deployment     │ 4,000.0          │ null               │ true        │ 1                   │ null                    │ null      │ null      │ null     │ null                │ null   │ always        │ true             │
├────────────────────────────────────────────────┼────────────────┼──────────────────┼────────────────────┼─────────────┼─────────────────────┼─────────────────────────┼───────────┼───────────┼──────────┼─────────────────────┼────────┼───────────────┼──────────────────┤
│ QmSqJEGHp1PcgvBYKFF2u8vhJZt8JTq18EV7mCuuZZiutX │ deployment     │ 5,000.0          │ null               │ true        │ 1                   │ null                    │ null      │ null      │ null     │ null                │ null   │ always        │ true             │
├────────────────────────────────────────────────┼────────────────┼──────────────────┼────────────────────┼─────────────┼─────────────────────┼─────────────────────────┼───────────┼───────────┼──────────┼─────────────────────┼────────┼───────────────┼──────────────────┤
│ QmW8Cbb2R4ZHWGsrYjNJKRjoKKcPeDTNK6rdipfQQaAhd6 │ deployment     │ 5,000.0          │ null               │ true        │ 1                   │ null                    │ null      │ null      │ null     │ null                │ null   │ always        │ true             │
├────────────────────────────────────────────────┼────────────────┼──────────────────┼────────────────────┼─────────────┼─────────────────────┼─────────────────────────┼───────────┼───────────┼──────────┼─────────────────────┼────────┼───────────────┼──────────────────┤
│ QmWq1pmnhEvx25qxpYYj9Yp6E1xMKMVoUjXVQBxUJmreSe │ deployment     │ 5,000.0          │ null               │ true        │ 1                   │ null                    │ null      │ null      │ null     │ null                │ null   │ always        │ true             │
└────────────────────────────────────────────────┴────────────────┴──────────────────┴────────

$graph indexer status --output=yaml


  url: http://localhost:7600/
  address: "0x........................"
  registered: true
  location: 9.115583896636963,10.751087665557861
endpoints:
  - name: service
    url: http://localhost:7600/
    status: up
    tests:
      - test: URL validation
        error: null
        possibleActions: []
        __typename: IndexerEndpointTest
      - test: http get http://localhost:7600/
        error: null
        possibleActions: []
        __typename: IndexerEndpointTest
  - name: status
    url: http://localhost:7600/status
    status: up
    tests:
      - test: URL validation
        error: null
        possibleActions: []
        __typename: IndexerEndpointTest
      - test: http post http://localhost:7600/status query="{ indexingStatuses {
          subgraph } }"
        error: null
        possibleActions: []
        __typename: IndexerEndpointTest
indexerDeployments:
  - subgraphDeployment: QmSqJEGHp1PcgvBYKFF2u8vhJZt8JTq18EV7mCuuZZiutX
    synced: true
    health: healthy
    fatalError: null
    node: default
    chains:
      - network: gnosis
        latestBlock:
          number: 24459209
          __typename: BlockPointer
        chainHeadBlock:
          number: 24525063
          __typename: BlockPointer
        earliestBlock:
          number: 14861343
          __typename: BlockPointer
        __typename: ChainIndexingStatus
    __typename: IndexerDeployment
  - subgraphDeployment: QmW8Cbb2R4ZHWGsrYjNJKRjoKKcPeDTNK6rdipfQQaAhd6
    synced: false
    health: healthy
    fatalError: null
    node: default
    chains:
      - network: gnosis
        latestBlock:
          number: 21342398
          __typename: BlockPointer
        chainHeadBlock:
          number: 24525063
          __typename: BlockPointer
        earliestBlock:
          number: 14735903
          __typename: BlockPointer
        __typename: ChainIndexingStatus
    __typename: IndexerDeployment
  - subgraphDeployment: QmWq1pmnhEvx25qxpYYj9Yp6E1xMKMVoUjXVQBxUJmreSe
    synced: true
    health: healthy
    fatalError: null
    node: default
    chains:
      - network: gnosis
        latestBlock:
          number: 24459209
          __typename: BlockPointer
        chainHeadBlock:
          number: 24525063
          __typename: BlockPointer
        earliestBlock:
          number: 18410654
          __typename: BlockPointer
        __typename: ChainIndexingStatus
    __typename: IndexerDeployment
  - subgraphDeployment: QmeVXKzGKSyfEQib4MzeZveJgDYJCYDMMHc1pPevWeSbsq
    synced: false
    health: healthy
    fatalError: null
    node: default
    chains:
      - network: gnosis
        latestBlock:
          number: 22629840
          __typename: BlockPointer
        chainHeadBlock:
          number: 24525063
          __typename: BlockPointer
        earliestBlock:
          number: 17890041
          __typename: BlockPointer
        __typename: ChainIndexingStatus
    __typename: IndexerDeployment
indexerAllocations: []
indexingRules:
  - identifier: QmWq1pmnhEvx25qxpYYj9Yp6E1xMKMVoUjXVQBxUJmreSe
    identifierType: deployment
    allocationAmount:
      type: BigNumber
      hex: "0x010f0cf064dd59200000"
    allocationLifetime: null
    autoRenewal: true
    parallelAllocations: 1
    maxAllocationPercentage: null
    minSignal: null
    maxSignal: null
    minStake: null
    minAverageQueryFees: null
    custom: null
    decisionBasis: always
    requireSupported: true
  - identifier: QmW8Cbb2R4ZHWGsrYjNJKRjoKKcPeDTNK6rdipfQQaAhd6
    identifierType: deployment
    allocationAmount:
      type: BigNumber
      hex: "0x010f0cf064dd59200000"
    allocationLifetime: null
    autoRenewal: true
    parallelAllocations: 1
    maxAllocationPercentage: null
    minSignal: null
    maxSignal: null
    minStake: null
    minAverageQueryFees: null
    custom: null
    decisionBasis: always
    requireSupported: true
  - identifier: QmSqJEGHp1PcgvBYKFF2u8vhJZt8JTq18EV7mCuuZZiutX
    identifierType: deployment
    allocationAmount:
      type: BigNumber
      hex: "0x010f0cf064dd59200000"
    allocationLifetime: null
    autoRenewal: true
    parallelAllocations: 1
    maxAllocationPercentage: null
    minSignal: null
    maxSignal: null
    minStake: null
    minAverageQueryFees: null
    custom: null
    decisionBasis: always
    requireSupported: true
  - identifier: QmeVXKzGKSyfEQib4MzeZveJgDYJCYDMMHc1pPevWeSbsq
    identifierType: deployment
    allocationAmount:
      type: BigNumber
      hex: "0xd8d726b7177a800000"
    allocationLifetime: null
    autoRenewal: true
    parallelAllocations: 1
    maxAllocationPercentage: null
    minSignal: null
    maxSignal: null
    minStake: null
    minAverageQueryFees: null
    custom: null
    decisionBasis: always
    requireSupported: true
  - identifier: global
    identifierType: group
    allocationAmount:
      type: BigNumber
      hex: "0x056bc75e2d63100000"
    allocationLifetime: null
    autoRenewal: true
    parallelAllocations: 1
    maxAllocationPercentage: null
    minSignal: null
    maxSignal: null
    minStake: null
    minAverageQueryFees: null
    custom: null
    decisionBasis: rules
    requireSupported: true

fordN commented 2 years ago

Hey @begetan & @papadritta 👋

It appears the indexer-agent is operating off of old or incorrect subgraph data. More specifically it looks like the agent is querying the network subgraph, seeing that there are no allocations for your indexer and then deciding to create them. In the status output you shared @papadritta, the indexerAllocations array is returning as empty.

Could you provide some detail around your network subgraph setup? What do you have set for network-subgraph-deployment and network-subgraph-endpoint?

papadritta commented 2 years ago

Hey @fordN I used old config set as network-subgraph-endpoint https://gateway.testnet.thegraph.com/network and now switch to the new network-subgraph-endpoint https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-goerli and add network-subgraph-deployment QmNhsC5jwSvwgYFdjBrjcc2YW8o7Xmo2WNZUHDukfRvzD4 and restart the agent. The graph-node version is v0.28.1

begetan commented 2 years ago

@fordN This is my config:

"INDEXER_AGENT_NETWORK_SUBGRAPH_ENDPOINT=https://gateway.testnet.thegraph.com/network"
"INDEXER_AGENT_NETWORK_SUBGRAPH_DEPLOYMENT=",

Network subgraph deployment is empty

papadritta commented 2 years ago

@begetan, you should change your api to network-subgraph-endpoint https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-goerli and delete ~~network-subgraph-deployment QmNhsC5jwSvwgYFdjBrjcc2YW8o7Xmo2WNZUHDukfRvzD4~~ because this subgraph broken and restart the agent

begetan commented 2 years ago

I set finally network-subgraph-endpointequal to https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-goerli

And left network-subgraph-deployment blank

Please be aware, that StakeSquid setup embeds broken default value in several place like here

I changed configuration to a single variable in .env file and replaced default value with this variable in docke-compose file:

.env:
NETWORK_SUBGRAPH_ENDPOINT=https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-goerli

docker-compose.yml:
INDEXER_SERVICE_NETWORK_SUBGRAPH_ENDPOINT: ${NETWORK_SUBGRAPH_ENDPOINT}
INDEXER_AGENT_NETWORK_SUBGRAPH_ENDPOINT: ${NETWORK_SUBGRAPH_ENDPOINT}
- subgraph_endpoint=${NETWORK_SUBGRAPH_ENDPOINT}

(repeated it three times around the whole config)

fordN commented 4 months ago

Closing this issue as it's old.

graphprotocol / indexer

indexer-agent: is trying to open allocations non stop on mips testnet #500