anza-xyz / agave

Web-Scale Blockchain for fast, secure, scalable, decentralized apps and marketplaces.
https://www.anza.xyz/
Apache License 2.0
430 stars 212 forks source link

Solana 1.18.8 hangs when deploying #1018

Open b1acKr0se opened 6 months ago

b1acKr0se commented 6 months ago

Problem

When running solana program deploy with solana v1.18.8, the process will hang indefinitely and not resign/resend transactions after a while. There's no 5 retries like previous versions.

⠲   1.6% | Resent 1132 transactions...              [block height 241935494; re-sign in 0 blocks] 

This causes the buffer account's seed phrase to never be printed, therefore the only option left is to close the buffer account through solana program close.

Proposed Solution

It should throw an error together with the seed phrase of the buffer account like it did before, so we know when to restart/resume the deploy.

b1acKr0se commented 6 months ago

Same with 1.18.11, even with high --max-sign-attempts, it will hang after the first block height becomes invalid.

valentinmadrid commented 6 months ago

Same issue here, and have also seen other people encountering it.

sjoerd-dijkstra commented 6 months ago

Same issue, across multiple versions and architectures. Don't know how to get past it..

b1acKr0se commented 6 months ago

Same issue, across multiple versions and architectures. Don't know how to get past it..

I had to fork the 1.14.29 version and made a change to the number of retry attempts. It finally worked after about 30 retries...

sjoerd-dijkstra commented 6 months ago

Same issue, across multiple versions and architectures. Don't know how to get past it..

I had to fork the 1.14.29 version and made a change to the number of retry attempts. It finally worked after about 30 retries...

Can confirm this works. Forked and changed the retry amount and it's not hanging. Thanks!

https://github.com/solana-labs/solana/blob/v1.14.29/client/src/tpu_client.rs#L224

phumaster commented 6 months ago

Same issue, across multiple versions and architectures. Don't know how to get past it..

I had to fork the 1.14.29 version and made a change to the number of retry attempts. It finally worked after about 30 retries...

It works. Thanks, you saved my life <3

bshumi commented 6 months ago

Even 30 retries didn't help. Seems, I should add prioritization fee to the old cli.

ad7090 commented 6 months ago

Same issue #35534

enzoampil commented 6 months ago

Making a fork, adding a timeout to send_and_confirm_transactions_in_parallel.rs (using time::timeout), and then applying retries on the same buffer account fixed it for me.

Athough it's really slow but did the job eventually (40 retries over 1.5 hours)

enzoampil commented 6 months ago

Working for me now with --use-rpc flag and helius + jito + ironforge RPC on 1.18.12

jacobcreech commented 6 months ago

CLI: 1.18.12

Verified it wasn't a swallowing of 429s by running solana program deploy target/deploy/deployment.so --with-compute-unit-price 1000 --max-sign-attempts 1000 with a known rated limited 429 endpoint, got thrown:

Error: HTTP status client error (429 Too Many Requests) for url XXXX

Found that hanging doesn't actually happen, but time between re-sign in 0 blocks and Blockhash expired. 997 retries remaining is measured in minutes.