Job error: Presumably has been uncled

[egodigitus]

Hi guys, trying to run through the initial tutorial of how to run a chainlink node. Was sending a job to my own node. Was calling the "requestEthereumChange" function on my own oracle contract. The job fails with the error in the log "job_error=TxHash 0xd44dac9c0c40b819d56df141a71f410138e45d465295923a2737fd3b98ff72b6 initiating run d97ddc0154f843fd9957b3d889e90ec5 not on main chain; presumably has been uncled"

I'm sure that metamask and therefore remix is on ropsten. I don't even have mainnet eth. Only ropsten testnet eth.

Here is the log message in context (below you'll find my setup and additional information)

2019-07-14T19:16:12Z [DEBUG] Received new head #5985045 (0x5b5315)              services/head_tracker.go:208 blockHash=0x27396426407379056f00752e3f9491e5244ab4571297f5c6c0d4372f90ea9f23 blockHeight=5985045 
2019-07-14T19:16:18Z [DEBUG] Received new head #5985046 (0x5b5316)              services/head_tracker.go:208 blockHash=0xbfed75bc51ff0d8612db1c51cffac0e2aef6c6875253867987b333ee326c1d9f blockHeight=5985046 
2019-07-14T19:16:18Z [DEBUG] Log for runlog initiator for job 132b20c5241c45799deac37024c4d56d services/subscription.go:109 blockNumber=5985046 job=132b20c5241c45799deac37024c4d56d logIndex=48 txHash=0xd44dac9c0c40b819d56df141a71f410138e45d465295923a2737fd3b98ff72b6 
2019-07-14T19:16:18Z [DEBUG] Received log from block #5985046 for address 0x2283767b07CF66779BEFA0a39B8a642237C261A8 for job 132b20c5241c45799deac37024c4d56d models/log_events.go:201 initiator={"id":4,"jobSpecId":"132b20c5241c45799deac37024c4d56d","type":"runlog","CreatedAt":"2019-07-14T18:11:09.406395651Z","params":{"time":null,"address":"0x2283767b07cf66779befa0a39b8a642237c261a8"}} job=132b20c5241c45799deac37024c4d56d log=5985046 topic0=0xd8d7ecc4800d25fa53ce0372f13a416d98907a7ef3d8d3bdd79cf4fe75529c65 
2019-07-14T19:16:18Z [DEBUG] New run triggered by runlog                        services/runs.go:45     creation_height=5985046 job=132b20c5241c45799deac37024c4d56d 
2019-07-14T19:16:18Z [DEBUG] Run cannot continue because it lacks sufficient confirmations services/runs.go:330    required_height=3 run=d97ddc0154f843fd9957b3d889e90ec5 
2019-07-14T19:16:18Z [DEBUG] Pausing run originally initiated by runlog         services/runs.go:410    creation_height=5985046 job=132b20c5241c45799deac37024c4d56d observed_height=5985046 run=d97ddc0154f843fd9957b3d889e90ec5 status=pending_confirmations 
2019-07-14T19:17:02Z [DEBUG] Received new head #5985047 (0x5b5317)              services/head_tracker.go:208 blockHash=0xcadc007645460a59daf728dffbdde5ca8484d76520581d1c1b478cda826fd5ff blockHeight=5985047 
2019-07-14T19:17:02Z [DEBUG] New head resuming run                              services/runs.go:152    creation_height=5985046 job=132b20c5241c45799deac37024c4d56d observed_height=5985046 run=d97ddc0154f843fd9957b3d889e90ec5 status=pending_confirmations 
2019-07-14T19:17:02Z [DEBUG] Run cannot continue because it lacks sufficient confirmations services/runs.go:330    required_height=3 run=d97ddc0154f843fd9957b3d889e90ec5 
2019-07-14T19:17:02Z [DEBUG] Pausing run originally initiated by runlog         services/runs.go:410    creation_height=5985046 job=132b20c5241c45799deac37024c4d56d observed_height=5985047 run=d97ddc0154f843fd9957b3d889e90ec5 status=pending_confirmations 
2019-07-14T19:17:16Z [DEBUG] Received new head #5985048 (0x5b5318)              services/head_tracker.go:208 blockHash=0x6dd17c523a5666bba1efd2cea872b170e1e913f47498422d6d244f5b864d8886 blockHeight=5985048 
2019-07-14T19:17:16Z [DEBUG] New head resuming run                              services/runs.go:152    creation_height=5985046 job=132b20c5241c45799deac37024c4d56d observed_height=5985047 run=d97ddc0154f843fd9957b3d889e90ec5 status=pending_confirmations 
2019-07-14T19:17:16Z [DEBUG] Pausing run originally initiated by runlog         services/runs.go:410    creation_height=5985046 job=132b20c5241c45799deac37024c4d56d job_error=TxHash 0xd44dac9c0c40b819d56df141a71f410138e45d465295923a2737fd3b98ff72b6 initiating run d97ddc0154f843fd9957b3d889e90ec5 not on main chain; presumably has been uncled observed_height=5985048 run=d97ddc0154f843fd9957b3d889e90ec5 status=errored 
2019-07-14T19:17:20Z [DEBUG] Received new head #5985049 (0x5b5319)              services/head_tracker.go:208 blockHash=0x9b2985abd82d8aae3c01fb53056f0063a32743cb7f81d5789704af71b67740c4 blockHeight=5985049 
2019-07-14T19:17:30Z [DEBUG] Received new head #5985050 (0x5b531a)              services/head_tracker.go:208 blockHash=0xa1fa7a40c7c64bbcfa1bbf22d4c1bbef4ebd5338d0c315577908ee0b82b0aa0e blockHeight=5985050 

System Information

Running Chainlink Node 0.6.3 at commit c19dc8da1e2ae90b79b8444c1ec59616b98a475b inside docker on ubuntu 18.04. Running geth v1.9.0 - connected to ropsten

Environment Variables

ACCOUNT_ADDRESS: 0xC26Ec5a4B14B9f53E56B55d3c4b56667BD5d5bCb ALLOW_ORIGINS: * BRIDGE_RESPONSE_URL: ETH_CHAIN_ID: 3 CLIENT_NODE_URL: http://localhost:6688 CHAINLINK_DEV: false DATABASE_TIMEOUT: 500ms ETH_URL: ws://172.17.0.2:8546 ETH_GAS_BUMP_THRESHOLD: 12 ETH_GAS_BUMP_WEI: 5000000000 ETH_GAS_PRICE_DEFAULT: 20000000000 JSON_CONSOLE: false LINK_CONTRACT_ADDRESS: 0x20fe562d797a42dcb3399062ae9546cd06f63280 EXPLORER_URL: LOG_LEVEL: debug LOG_TO_DISK: true MINIMUM_CONTRACT_PAYMENT: 1.000000000000000000 MINIMUM_REQUEST_EXPIRATION: 300 MIN_INCOMING_CONFIRMATIONS: 3 MIN_OUTGOING_CONFIRMATIONS: 2 ORACLE_CONTRACT_ADDRESS: CHAINLINK_PORT: 6688 REAPER_EXPIRATION: 240h0m0s ROOT: /chainlink SESSION_TIMEOUT: 15m0s CHAINLINK_TLS_HOST: CHAINLINK_TLS_PORT: 0

Additional Information

EthUnit256 JobSpec with ID 132b20c5241c45799deac37024c4d56d:

{
  "initiators": [
    {
      "type": "runlog",
      "params": {
        "address": "0x2283767b07cf66779befa0a39b8a642237c261a8"
      }
    }
  ],
  "tasks": [
    {
      "type": "httpget",
      "confirmations": null,
      "params": {
      }
    },
    {
      "type": "jsonparse",
      "confirmations": null,
      "params": {
      }
    },
    {
      "type": "multiply",
      "confirmations": null,
      "params": {
      }
    },
    {
      "type": "ethuint256",
      "confirmations": null,
      "params": {
      }
    },
    {
      "type": "ethtx",
      "confirmations": null,
      "params": {
      }
    }
  ],
  "startAt": null,
  "endAt": null
}

• The Error Log of the job in the GUI is only stating "Errored"
• Oracle contract address: 0x061e493db8efdb9fcf623106470aa6897c689c6d
• Transaction ID: 0xd44dac9c0c40b819d56df141a71f410138e45d465295923a2737fd3b98ff72b6

[lilianwen]

i meet the same problem.

[se3000]

This is expected behavior, that's why it is a debug level log message. Reorgs happen regularly on every network, including Ropsten. If a transaction is in a block that is uncled, Chainlink will detect that and only respond to the request on the winning chain.

Closing this issue, as it is the desired behavior. If you see this happen, and a request is responded to multiple times or not responded to at all, feel free to reopen.

[roheat]

I am also getting "Errored" in the GUI due to the same issue. How do I fix it? @egodigitus let me know if you found the solution.

[egodigitus]

@roheat I haven't tried it, cause I lost interest in chainlink, but a befriended eth dev gave me a hint: Try to increase the block confirmations needed. There are the two env variables MIN_INCOMING_CONFIRMATIONS and MIN_OUTGOING_CONFIRMATIONS. Try something like 12

[gretzke]

@roheat I don't think this is an issue. I'm currently running a cron job calling my oracle every 15 minutes. I got the same error, but a successful run immediately after it. So my guess is that the job still completed successfully