search

solana-labs / solana

Web-Scale Blockchain for fast, secure, scalable, decentralized apps and marketplaces.
https://solanalabs.com
Apache License 2.0
13.35k stars 4.35k forks source link

Query: Deploying hello world to solana-test-validator on M1 fails #18033

Closed fdeantoni closed 2 years ago

fdeantoni commented 3 years ago

Problem

I'm running Solana master with the example Hello World program and on OSX x86 things run as expected. I can deploy the hello world program to the test validator. However, when doing the exact same on M1, it fails at the "Finalize transaction" step:

$ solana program deploy dist/program/helloworld.so
===================================================================
Recover the intermediate account's ephemeral keypair file with
`solana-keygen recover` and the following 12-word seed phrase:
===================================================================
spring kick mail one box meat august educate car ghost orient woman
===================================================================
To resume a deploy, pass the recovered keypair as
the [PROGRAM_ADDRESS_SIGNER] argument to `solana deploy` or
as the [BUFFER_SIGNER] to `solana program deploy` or `solana write-buffer'.
Or to recover the account's lamports, pass it as the
[BUFFER_ACCOUNT_ADDRESS] argument to `solana program close`.
===================================================================
Error: Deploying program failed: unable to confirm transaction. This can happen in situations such as transaction expiration and insufficient fee-payer funds

I was hoping if anybody would have a hint where I should look to try and resolve this.

This is what I did to get my environment started:

  1. Set up BPF SDK 

    $ git clone https://github.com/solana-labs/solana
$ export PROJECT_ROOT=`pwd`/solana
$ cd $PROJECT_ROOT/sdk/cargo-build-bpf
$ cargo install --path .
$ cd $PROJECT_ROOT
$ ln -s $PROJECT_ROOT/sdk ~/.cargo/bin/sdk

  2. Build the CLI and run configure

    $ cd $PROJECT_ROOT/cli
$ cargo build
$ cd $PROJECT_ROOT/keygen
$ cargo build
$ $PROJECT_ROOT/target/debug/solana  config set --url localhost
$ $PROJECT_ROOT/target/debug/solana-keygen new

  3. Run the test validator

    $ cd $PROJECT_ROOT/validator
$ ./solana-test-validator

  4. Build hello world Edit the example-helloworld/src/program-rust/Cargo.toml to match $PROJECT_ROOT deps, and then:

    $ git clone https://github.com/solana-labs/example-helloworld
$ cd example-helloworld
$ npm install
$ npm run build:program-rust

Everything compiles ok and seems to run fine except for the "Finalize transaction" part which fails on M1. In the validator.log I see no errors, only a warning:

WARN  solana_core::window_service] Window does not seem to be receiving data. Ensure port configuration is correct...

I do not see this warning when running on OSX x86.

Any hints where I should look to try and fix this?

Proposed Solution

None - just a query.

neildotvn commented 3 years ago

Any update on this? I have the same problem.

gxxcastillo commented 3 years ago

Similar issue, I'm on a Mac with an M1 Chip. I've tried different demos and all of them have the same issue when I try to deploy my program. Basically it just gets stuck at [0/1] Finalizing transaction...

validator.logs has this being called repeatedly: solana_rpc::send_transaction_service] Retrying transaction: when I grep using my transaction id.

I've tried building + deploying with my terminal running Rosetta and run into the same problem regardless, so not sure if this is related.

gxxcastillo commented 3 years ago

I found my issue: I had installed almost everything under Rosetta. Checking my rustup toolchain via rustup toolchain list I discovered I had somehow managed to install rust for arm64 instead of x86. After reinstalling everything, I've confirmed my default toolchain is now stable-x86_64-apple-darwin and everything is now building 😃😌

monomadic commented 3 years ago

@gxxcastillo okay but that isn't really fixing the issue it's avoiding it. I'm getting the same issue on M1 - funny thing is, it was deploying perfectly earlier - but I can't confirm if this was the x86 version.

I get the feeling this occurs when part of your toolchain is x86 and some is arm, but can't confirm yet. I'll keep digging.

Qumeric commented 3 years ago

I also use M1 mac and I have

Error: Account address has insufficient funds for spend (0.4265436 SOL) + fee (0.001355 SOL)

Does it look like the same issue? 🤔

Although npm run build:program-c causes the same error.

gxxcastillo commented 3 years ago

@Qumeric, make sure 1) You're connected to the right cluster and 2) That you've got enough SOL in your account

Qumeric commented 3 years ago

I have found my issue. I built Solana from sources (because otherwise, I faced another M1 related issue) and used ./run.sh as it is stated in https://github.com/solana-labs/solana/blob/master/README.md. However, what helped me is to run ./solana/target/debug/solana-test-validator instead.

(Also I tried to do solana airdrop 100 previously but it stuck on [0/1] Finalizing transaction... as described in some previous reply)

artursapek commented 3 years ago

pretty sad the fastest blockchain doesn't work on the fastest laptops :(

Krishna-Psftware commented 3 years ago

Similar issue even am facing while requesting airdrop

Error: unable to confirm transaction. This can happen in situations such as transaction expiration and insufficient fee-payer funds

Krishna-Psftware commented 3 years ago

I have found my issue. I built Solana from sources (because otherwise, I faced another M1 related issue) and used ./run.sh as it is stated in https://github.com/solana-labs/solana/blob/master/README.md. However, what helped me is to run ./solana/target/debug/solana-test-validator instead.

(Also I tried to do solana airdrop 100 previously but it stuck on [0/1] Finalizing transaction... as described in some previous reply)

Krishna-Psftware commented 3 years ago

I have found my issue. I built Solana from sources (because otherwise, I faced another M1 related issue) and used ./run.sh as it is stated in https://github.com/solana-labs/solana/blob/master/README.md. However, what helped me is to run ./solana/target/debug/solana-test-validator instead.

(Also I tried to do solana airdrop 100 previously but it stuck on [0/1] Finalizing transaction... as described in some previous reply)

Did you resolve this issue?

crmanik commented 3 years ago

Hello, I tried following:

But still I get this damn error:

error: unable to confirm transaction. This can happen in situations such as transaction expiration and insufficient fee-payer funds

I'm really stuck, any pointers are how to move forward?

dcolley commented 3 years ago

There is a Docker image dedicated to mac-m1 - https://github.com/metaspan/solana-docker-mac-m1 I have the same problem and my logs show the following:

[2021-09-30T08:56:11.948913055Z INFO  solana_rpc::send_transaction_service] Retrying transaction: 41yweJXm4inkN7BZ5TfwyHV3FUYaZ84YMmMp386BNKgSnLtn9Jiha8YCABNsfJjoDSnyYYAkZLMJdZ8gAUx41aC
thread 'solana-banking-stage-tx' panicked at 'JIT is only supported on x86_64', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/solana_rbpf-0.2.11/src/jit.rs:738:13
stack backtrace:
   0: std::panicking::begin_panic
   1: solana_rbpf::jit::JitCompiler::new
   2: solana_rbpf::jit::JitProgram<E,I>::new
   3: <solana_rbpf::elf::EBpfElf<E,I> as solana_rbpf::vm::Executable<E,I>>::jit_compile
   4: solana_bpf_loader_program::create_executor
   5: solana_bpf_loader_program::process_loader_upgradeable_instruction
   6: solana_bpf_loader_program::process_instruction_common
   7: solana_runtime::message_processor::MessageProcessor::process_instruction
   8: solana_runtime::message_processor::MessageProcessor::process_message
   9: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
  10: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
  11: solana_runtime::bank::Bank::load_and_execute_transactions
  12: solana_core::banking_stage::BankingStage::process_and_record_transactions
  13: solana_core::banking_stage::BankingStage::process_packets_transactions
  14: solana_core::banking_stage::BankingStage::process_packets
  15: solana_core::banking_stage::BankingStage::process_loop
dcolley commented 3 years ago

Seems this is linked: https://github.com/qmonnet/rbpf/issues/48

codenjobs commented 3 years ago

If you are ok to use the terminal with Intel architecture, you can refer to this blog post How to make Solana Test Validator work with a Macbook with M1 chip instead.

amalcaraz commented 3 years ago

I have the same issue with an M1 mac and found a workaround starting the validator with the "--bpf-program" option. Not doing the deployment process, but at least I could have the program in my testnet and keep working with it

solana-test-validator --bpf-program ADDRESS target/deploy/PROGRAM.so
newhouseb commented 3 years ago

From perusing the discord, I've figured out what was up (as alluded to in other threads):

  1. Adding --log to the test-validator command and watching the deploy, you'll see the following 
    [2021-11-08T00:40:59.691095000Z INFO  solana_core::heaviest_subtree_fork_choice] marking fork starting at: (4505, DAmaeMFkDVRaeyuD9HDfvCsy4JDbJDui1YHvrzMsH5Ao) valid candidate
thread 'solana-banking-stage-tx' panicked at 'JIT is only supported on x86_64', /Users/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/solana_rbpf-0.2.11/src/jit.rs:738:13
stack backtrace:
  2. As you can see: the BPF JIT is not going to work on arm64. Fortunately you can run things without the JIT by adding --no-bpf-jit to the test-validator command and then you'll be good to go.
fanatid commented 3 years ago

https://github.com/solana-labs/solana/pull/21215 should help to run test validator without explicit --no-bpf-jit flag.

newhouseb commented 3 years ago

Also heads up that as you try to get the hello world to actually work you'll run into further issues due to BPF parameter alignment as mentioned here: https://github.com/solana-labs/solana/issues/21269

nphenley commented 3 years ago

I have found my issue. I built Solana from sources (because otherwise, I faced another M1 related issue) and used ./run.sh as it is stated in https://github.com/solana-labs/solana/blob/master/README.md. However, what helped me is to run ./solana/target/debug/solana-test-validator instead. (Also I tried to do solana airdrop 100 previously but it stuck on [0/1] Finalizing transaction... as described in some previous reply)

Did you find a solution to this?

chihkaiyu commented 3 years ago

I build Solana tool suite from source and add --no-bpf-jit flag to solana-test-validator. Now at least I can deploy the example program to local validator.

Before that, I can't even run solana-test-validator. Using the Solana CLI which is built from source, the CLI works but then it's stuck when deploying the program.

$ solana program deploy dist/program/helloworld.so                                                                                                                                                          

===========================================================================                           
Recover the intermediate account's ephemeral keypair file with                                        
`solana-keygen recover` and the following 12-word seed phrase:                                        
===========================================================================                           
gas tiger plastic knife speak uncle monkey rebel boat suspect agree concert                           
===========================================================================                           
To resume a deploy, pass the recovered keypair as the                                                 
[BUFFER_SIGNER] to `solana program deploy` or `solana write-buffer'.                                  
Or to recover the account's lamports, pass it as the                                                  
[BUFFER_ACCOUNT_ADDRESS] argument to `solana program close`.                                          
===========================================================================                           
Error: Deploying program failed: unable to confirm transaction. This can happen in situations such as transaction expiration and insufficient fee-payer funds

I resume the failed deployment but it still stuck and finally reach maximum retries.

$ solana program deploy --buffer ~/.config/solana/id.json ./dist/program/helloworld.so 
Blockhash expired. 5 retries remaining             
Blockhash expired. 4 retries remaining             
Blockhash expired. 3 retries remaining             
Blockhash expired. 2 retries remaining             
Blockhash expired. 1 retries remaining             
Error: Data writes to account failed: Custom error: Max retries exceeded
diginator commented 3 years ago

Did you get through this? $ solana program deploy and $anchor deplay are both getting stuck at: Blockhash expired. 5 retries remaining
Blockhash expired. 4 retries remaining
Blockhash expired. 3 retries remaining
Blockhash expired. 2 retries remaining
Blockhash expired. 1 retries remaining
Error: Data writes to account failed: Custom error: Max retries exceeded

Vkzem commented 2 years ago

Hello, I tried following:

  • installed rust in rosetta terminal.
  • compiled Solana tools from source
  • Able to start Solana-test-validator
  • Confirmed endpoint is right.

But still I get this damn error:

error: unable to confirm transaction. This can happen in situations such as transaction expiration and insufficient fee-payer funds

I'm really stuck, any pointers are how to move forward?

did you manage to solve this problem? also facing the same even when trying to do a devnet airdrop

diginator commented 2 years ago

So, I think might be a network error. I switched to a different network and since the devnet wallet is currently drained, I tried using testnet and it totally worked. Btw, I tried using testnet earlier when was stuck at the deployment but it wouldn't work. Changing the network connection is what got it working for me.

Vkzem commented 2 years ago

thanks mate, I think you're exactly right - plus looks like the devnet issue is resolved now, too

n0x1m commented 2 years ago

I have a reverse setup and similar issue on v1.9.2, I build on a Mac x86_64 and deploy to a Raspberry Pi test validator, it never succeeds but retries to send transactions and then times out, requiring me to close the buffer.

Blockhash expired. 5 retries remaining
Blockhash expired. 4 retries remaining
Blockhash expired. 3 retries remaining
Blockhash expired. 2 retries remaining
Blockhash expired. 1 retries remaining
Error: Data writes to account failed: Custom error: Max retries exceeded

How can I debug this?

diginator commented 2 years ago

This is a network issue and you're probably facing this now because the Solana network is down. Check at status.solana.com and try again later. If it still happens, switch your network.

t-nelson commented 2 years ago

This is a network issue and you're probably facing this now because the Solana network is down. Check at status.solana.com and try again later. If it still happens, switch your network.

They're obviously running a private cluster. None of the public clusters will influence it, congestion or otherwise

I have a reverse setup and similar issue on v1.9.2, I build on a Mac x86_64 and deploy to a Raspberry Pi test validator, it never succeeds but retries to send transactions and then times out, requiring me to close the buffer.

Blockhash expired. 5 retries remaining
Blockhash expired. 4 retries remaining
Blockhash expired. 3 retries remaining
Blockhash expired. 2 retries remaining
Blockhash expired. 1 retries remaining
Error: Data writes to account failed: Custom error: Max retries exceeded

How can I debug this?

Have you tried to deploy from the rPi directly to rule out network configuration issues.

n0x1m commented 2 years ago

Have you tried to deploy from the rPi directly to rule out network configuration issues.

if I scp the binary to the rpi I can deploy it from there without issues. Networking is fine otherwise. It appears some issue with the tooling crossing the architectures (?) but I don't know how to debug this further.

dcolley commented 2 years ago

After upgrade to latest tools I have a different issue, and google leads me back here... (I forgot about this issue).

Running Ubuntu 18.04 arm64 in Docker on Mac M1

root@04598e084504:~/test# anchor --version
anchor-cli 0.22.1
root@04598e084504:~/test# rustc --version
rustc 1.59.0 (9d1b2106e 2022-02-23)
root@04598e084504:~/test# rustup --version
rustup 1.24.3 (ce5817a94 2021-05-31)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.59.0 (9d1b2106e 2022-02-23)`
root@04598e084504:~/test# solana --version
solana-cli 1.9.9 (src:devbuild; feat:3246413280)
root@04598e084504:~/test# yarn --version
1.22.17
root@04598e084504:~/test# uname -a
Linux 04598e084504 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
$ anchor init test
yarn install v1.22.17
warning package.json: No license field
info No lockfile found.
warning No license field
[1/4] Resolving packages...
warning @project-serum/anchor > @solana/web3.js > rpc-websockets > circular-json@0.5.9: CircularJSON is in maintenance only, flatted is its successor.
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > ts-mocha@8.0.0" has incorrect peer dependency "mocha@^3.X.X || ^4.X.X || ^5.X.X || ^6.X.X || ^7.X.X || ^8.X.X".
[4/4] Building fresh packages...
success Saved lockfile.
Done in 16.56s.
test initialized
$ cd test
$ anchor build
BPF SDK: /opt/solana-1.9.9/bin/sdk/bpf
cargo-build-bpf child: rustup toolchain list -v
cargo-build-bpf child: cargo +bpf build --target bpfel-unknown-unknown --release
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

More info:

# rustup toolchain list -v
stable-aarch64-unknown-linux-gnu (default)  /root/.rustup/toolchains/stable-aarch64-unknown-linux-gnu
bpf /opt/solana-1.9.9/bin/sdk/bpf/dependencies/bpf-tools/rust

It seems related to the toolchain for aarch64

nathanleclaire commented 2 years ago

Same issue here, can't get test validator to successfully receive programs in docker container running on m1 mac. Native mac binary works fine.

nathanleclaire commented 2 years ago

Deploys are working fine for me now with solana-test-validator running in a container on M1 using this Dockerfile and this Docker run command.

shawnwang0715 commented 2 years ago 
$ cd test
$ anchor build
BPF SDK: /opt/solana-1.9.9/bin/sdk/bpf
cargo-build-bpf child: rustup toolchain list -v
cargo-build-bpf child: cargo +bpf build --target bpfel-unknown-unknown --release
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

More info:

# rustup toolchain list -v
stable-aarch64-unknown-linux-gnu (default)    /root/.rustup/toolchains/stable-aarch64-unknown-linux-gnu
bpf   /opt/solana-1.9.9/bin/sdk/bpf/dependencies/bpf-tools/rust

It seems related to the toolchain for aarch64

@dcolley I have the same issue of qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory in docker on M1 Mac. Did you find out the solution?

dcolley commented 2 years ago

@shawnwang0715 no, I switched to ubuntu on x64 instead

shawnwang0715 commented 2 years ago

@shawnwang0715 no, I switched to ubuntu on x64 instead

Could you please describe how do you build on ubuntu x64? I used your dockerfile and tried to run cargo build-bpf inside the container but still failed. More details here: https://github.com/metaspan/solana-docker-mac-m1/issues/4

dcolley commented 2 years ago

@shawnwang0715 In my experience, arm64 (incl. Mac M1) is still too niche for many projects - especially those working with Rust. The low-level dependencies aren't available and I found myself debugging too long to try to get it working. After 40+ hours (over some weeks) I pulled the plug and got a £500 server. I am running ubuntu x64 on bare-metal. Just follow the instructions for ubuntu as per the docs.

steveluscher commented 2 years ago

I'm going to close this issue, but in doing so we're going to lose all of this valuable investigation and solution-finding.

Can someone please re-post the question and a summary of the answer to https://solana.stackexchange.com/questions/ask so that people who have these sorts of problems in the future can find their way to it? Thanks!