search

drift-labs / gateway

Self hosted API gateway to easily interact with Drift V2 Protocol
Apache License 2.0
22 stars 12 forks source link

Order placemenet fails even after multiple retries #88

Open satwikkansal opened 6 days ago

satwikkansal commented 6 days ago 
[2024-11-06T09:44:50Z INFO  gateway] 149.102.254.86 | 200 | GET /v2/markets HTTP/1.1 | (0.203708ms)
[2024-11-06T09:44:51Z INFO  gateway] 149.102.254.86 | 200 | GET /v2/markets HTTP/1.1 | (0.159284ms)
[2024-11-06T09:44:52Z INFO  gateway] accepted Ws connection: 149.102.254.86:35465
[2024-11-06T09:44:52Z INFO  gateway] subscribing to events for: 0
[2024-11-06T09:45:12Z INFO  gateway] 149.102.254.86 | 200 | POST /v2/orders?subAccountId=0 HTTP/1.1 | (70.208040ms)
[2024-11-06T09:45:44Z INFO  gateway] 149.102.254.86 | 200 | GET /v2/transactionEvent/2bqPN6Jk88kC7ZnE3JRZkHcYVBkBRQk7kUq3UfkMYLkcDNBGxuGEW5PvNCZNnBuwbaDsiKZ9NsdQ6KrqbGJqcwWC?subAccountId=0 HTTP/1.1 | (223.196969ms)
[2024-11-06T09:45:47Z INFO  gateway] 149.102.254.86 | 200 | POST /v2/orders?subAccountId=0 HTTP/1.1 | (550.595609ms)
[2024-11-06T09:46:19Z WARN  gateway] failed to get transaction: Error { request: Some(GetTransaction), kind: SerdeJson(Error("invalid type: null, expected struct EncodedConfirmedTransactionWithStatusMeta", line: 0, column: 0)) }, tx_error: None
[2024-11-06T09:46:19Z INFO  gateway] 149.102.254.86 | 404 | GET /v2/transactionEvent/4YPAB65LvSqtFp1tb3dVamDVFZjDsi9R1ZUzhMgWNjLQNV2VeJ1bzySbGPoQaccHPhRMuPoin6vjVNe1J1LdnTfW?subAccountId=0 HTTP/1.1 | (281.749459ms)
[2024-11-06T09:46:21Z INFO  gateway] 149.102.254.86 | 200 | POST /v2/orders?subAccountId=0 HTTP/1.1 | (557.742885ms)
[2024-11-06T09:46:53Z WARN  gateway] failed to get transaction: Error { request: Some(GetTransaction), kind: SerdeJson(Error("invalid type: null, expected struct EncodedConfirmedTransactionWithStatusMeta", line: 0, column: 0)) }, tx_error: None
[2024-11-06T09:46:53Z INFO  gateway] 149.102.254.86 | 404 | GET /v2/transactionEvent/4HEfeuXhuFzh3o5PDb3ZbLE5UfREqZAFkx56DVxBSar6nr1b5PKTh4VEWgTFRuhABhJ1KHKgDoLiuGgKbokM4aKt?subAccountId=0 HTTP/1.1 | (255.769429ms)
[2024-11-06T09:46:56Z INFO  gateway] 149.102.254.86 | 200 | POST /v2/orders?subAccountId=0 HTTP/1.1 | (567.745236ms)
[2024-11-06T09:47:28Z INFO  gateway] 149.102.254.86 | 200 | GET /v2/transactionEvent/RNLgwbt5jgmNsdbU6WxLPoAQsmawjpUVDF3F3mMV3ThcNuMYpQ5juYetW3bnXhyQa4a5xaS2PyTdNq2YJ24ituT?subAccountId=0 HTTP/1.1 | (228.263934ms)
[2024-11-06T09:47:32Z INFO  gateway] 149.102.254.86 | 200 | POST /v2/orders?subAccountId=0 HTTP/1.1 | (575.731289ms)

[2024-11-06T09:48:03Z WARN  gateway] failed to get transaction: Error { request: Some(GetTransaction), kind: SerdeJson(Error("invalid type: null, expected struct EncodedConfirmedTransactionWithStatusMeta", line: 0, column: 0)) }, tx_error: None
[2024-11-06T09:48:03Z INFO  gateway] 149.102.254.86 | 404 | GET /v2/transactionEvent/5m2qonhcWzLMRmtCEWts2ZTKtnpCmoBfXr4jQAmkwdnVzM4C9GhsSied22nVLWU9mkPenxhkyBcjSjQGsytr42oM?subAccountId=0 HTTP/1.1 | (305.001184ms)

[2024-11-06T09:48:07Z INFO  gateway] 149.102.254.86 | 200 | POST /v2/orders?subAccountId=0 HTTP/1.1 | (590.738517ms)
jordy25519 commented 6 days ago

confirming this is on latest commit/release?

satwikkansal commented 6 days ago

yes, I'm using the image available at ghcr.io/drift-labs/gateway

jordy25519 commented 6 days ago

from the logs it looks like the txs aren't being included in blocks by Solana validators. This can happen during busy network times mostly due to competitive tx fees.

for now I recommend your script to retry the tx if it still returns 404s after atleast 15s or can retry more aggressively with something like cancelAndPlace to ensure it doesn't place multiple orders

another option is set a higher CU price per request: https://github.com/drift-labs/gateway?tab=readme-ov-file#cu-price--limits when your script encounters more 404s

also related #67

utgarda commented 6 days ago

@jordy25519 every Gateway version has been doing the same since we first tried it, and we used to place orders with ridiculously high CU prices and different RPC providers too. Most order placing want a few retries, I hardly ever see orders that fly on the first attempt. What can I provide for diagnostics?

jordy25519 commented 6 days ago

also try starting gateway with the --skip-tx-preflight flag.

@jordy25519 every Gateway version has been doing the same since we first tried it, and we used to place orders with ridiculously high CU prices and different RPC providers too. Most order placing want a few retries, I hardly ever see orders that fly on the first attempt. What can I provide for diagnostics?

increasing chances of tx landing is a known challenge on Solana. Besides CU prices the options are basically retry the tx and sending to multiples RPC providers simultaneously, will add support for these things in next gateway update.

jordy25519 commented 2 days ago

please see new 1.2.0 release