search

smartcontractkit / chainlink

node of the decentralized oracle network, bridging on and off-chain computation
https://chain.link
Other
6.95k stars 1.69k forks source link

[NODE] Node API sometimes returns tasks in wrong order #2017

Closed bbenligiray closed 3 years ago

bbenligiray commented 4 years ago

I use this script to add multiple bridges and job specs using the node API. After I add the job specs, I fetch them using /v2/specs to validate that they are added correctly. The returned tasks are sometimes randomly rotated (for example Task4->Task1->Task2->Task3) or in some cases mirrored (Task4->Task3->Task2->Task1). Browsing the database, the task orders look correct and everything seems fine on the node GUI.

This is the setup that I can replicate the mirroring problem reliably but it is observed in a variety of setups, including AWS RDS (but not with larger instances for some reason) and a self-hosted database, but never with local sqlite.

Basic Information

Environment Variables

ACCOUNT_ADDRESS: 0x9A4EDF4f045cB14d4c789F9217FDC2305E81FF72
ALLOW_ORIGINS: *
BRIDGE_RESPONSE_URL: 
ETH_CHAIN_ID: 3
CLIENT_NODE_URL: http://localhost:6688
DATABASE_TIMEOUT: 
CHAINLINK_DEV: false
ETH_URL: wss://ropsten.infura.io/ws
ETH_GAS_BUMP_THRESHOLD: 12
ETH_GAS_BUMP_WEI: 5000000000
ETH_GAS_PRICE_DEFAULT: 20000000000
EXPLORER_URL: 
JSON_CONSOLE: true
LINK_CONTRACT_ADDRESS: 0x20fe562d797a42dcb3399062ae9546cd06f63280
LOG_LEVEL: debug
LOG_SQL_MIGRATIONS: true
LOG_SQL: true
LOG_TO_DISK: true
MAX_RPC_CALLS_PER_SECOND: 500
MINIMUM_CONTRACT_PAYMENT: 1.000000000000000000
MINIMUM_REQUEST_EXPIRATION: 300
MIN_INCOMING_CONFIRMATIONS: 3
MIN_OUTGOING_CONFIRMATIONS: 2
ORACLE_CONTRACT_ADDRESS: 
CHAINLINK_PORT: 6688
REAPER_EXPIRATION: 240h0m0s
REPLAY_FROM_BLOCK: -1
ROOT: /chainlink
SESSION_TIMEOUT: 15m0s
CHAINLINK_TLS_HOST: 
CHAINLINK_TLS_PORT: 0
CHAINLINK_TLS_REDIRECT: false
CHAINLINK_TX_ATTEMPT_LIMIT: 10

Steps to Reproduce EDIT: See my comment below.

Set up a Chainlink node wherever and a default Digital Ocean postgresql database (1 GB RAM / 1vCPU / 10 GB Disk / Primary only / NYC1 - PostgreSQL 11). Use this script to add the bridges and specs attached (bridges-specs.zip). The script removes and re-adds the job specs with disordered tasks (we were using this workaround because we used to encounter this issue <1 in 1000 jobs in an apparently completely stochastic pattern). With the setup described above, the tasks are always returned in the mirrored order so the script should get stuck in an infinite loop of retries. Although the first task is an external adapter in the example, the same thing happens with jobs composed of only core adapters.

bbenligiray commented 4 years ago

Here is a more minimal script that demonstrates the problem on the same setup without adding any bridges.

test.zip

j16r commented 4 years ago

Tracking here: https://www.pivotaltracker.com/story/show/170679043

topliceanu commented 4 years ago

Hi @bbenligiray. I'm having a crack at this with the caveat that I'm really new to the project. I wrote two tests to try to reproduce the error:

  1. the storage layer retrieves a Job's tasks in the correct order
  2. the api layer retrieves a Job's tasks in the correct order. This should reproduce the logic in the script you sent. Am I missing anything?

The big difference I can see is that the job you are creating has an initiator. I can't see why that would make a difference but I'll add the initiator to the test to make sure.

topliceanu commented 4 years ago

I added the initiator and I got the same result!

bbenligiray commented 4 years ago

Hi @topliceanu , are you saying that you can't reproduce the error?

PatrickAlphaC commented 3 years ago

Closing due to merged PR to fix this issue