Open acheroncrypto opened 1 year ago
Do you have some details on which specific instructions are throwing it? And is it always the same instructions on each program that errors?
For auction-house example, did you reproduce the issue with the client https://github.com/armaniferrante/auction-house/blob/armani/pda/tests/auction-house.ts ? I wonder if I can get it just using this repo as starting point
For auction-house example, did you reproduce the issue with the client https://github.com/armaniferrante/auction-house/blob/armani/pda/tests/auction-house.ts ? I wonder if I can get it just using this repo as starting point
Yes. Switching back to 1.14.18
makes it work without any modifications to program/client code https://github.com/coral-xyz/anchor/blob/383e44078811fdf26dc73a4ad834b812c7b5636e/.github/workflows/reusable-tests.yaml#L419.
I've run into this issue in Anchor CI and the easiest way to reproduce it would be to clone Anchor and run the mentioned tests(they are submodules). Test workflow is useful to check the commands to run.
Will try to check this out this week
@acheroncrypto here is a simple program to reproduce it:
use anchor_lang::prelude::*;
use anchor_spl::token::{self, Mint, TokenAccount};
declare_id!("xxxx");
const METADATA_ACCOUNT_PREFIX: &str = "xxxx";
#[program]
pub mod real_program {
use super::*;
pub fn create_metadata_account(ctx: Context<CreateMetadataAccount>, ) -> Result<()> {
Ok(())
}
}
#[derive(Accounts)]
pub struct CreateMetadataAccount<'info> {
/// CHECK
#[account(mut, signer)]
pub owner: AccountInfo<'info>,
pub mint: Account<'info, Mint>,
#[account(
init,
payer = owner,
seeds = [METADATA_ACCOUNT_PREFIX.as_bytes(), mint.key().as_ref()],
token::mint = mint,
token::authority = owner,
bump,
)]
pub real_metadata_account: Box<Account<'info, TokenAccount>>,
/// CHECK
#[account(address = anchor_spl::token::ID)]
pub token_program: AccountInfo<'info>,
/// CHECK
pub system_program: AccountInfo<'info>,
pub rent: Sysvar<'info, Rent>,
}
@acheroncrypto could you provide details about reproducing this run wtih auction-house? I've tried anchor test
(never used anchor before), it prints:
Warning: cargo-build-bpf is deprecated. Please, use cargo-build-sbf
cargo-build-bpf child: /home/klykov/.local/share/solana/install/active_release/bin/cargo-build-sbf --arch bpf
But it in the validator log program is not mentioned.
@KirillLykov you can see this run. It shows exactly what commands to run to reproduce it.
From the same CI logs:
1) auction-house
Creates an NFT mint:
Error: failed to send transaction: Transaction simulation failed: Error processing Instruction 0: Program failed to complete
at Connection.sendEncodedTransaction (/home/runner/work/anchor/anchor/ts/node_modules/@solana/web3.js/src/connection.ts:5646:13)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Connection.sendRawTransaction (/home/runner/work/anchor/anchor/ts/node_modules/@solana/web3.js/src/connection.ts:5605:20)
at sendAndConfirmRawTransaction (/home/runner/work/anchor/anchor/ts/packages/anchor/src/provider.ts:370:21)
at AnchorProvider.sendAndConfirm (/home/runner/work/anchor/anchor/ts/packages/anchor/src/provider.ts:160:14)
This error message is not very helpful but if you get the transaction logs, you will see Invoked an instruction with data that is too large
error.
Note that the first test is a setup step that doesn't even use the auction-house
program.It uses this program which doesn't use Anchor. If you build that program with an older Solana version, the first test(Creates an NFT mint
) passes but this time, Creates an auction house
test fails. Both of the tests fail with Invoked an instruction with data that is too large
error.
@effeaucarre could you provide also a client to execute this program?
I was getting the above error so when i downgraded to 1.14.18, i am getting the following error. ( anchor version: 0.27.0 )
Warning: cargo-build-bpf is deprecated. Please, use cargo-build-sbf
cargo-build-bpf child: /Users/dhruvjain/.local/share/solana/install/active_release/bin/cargo-build-sbf --arch bpf
error: package `toml_edit v0.19.10` cannot be built because it requires rustc 1.64.0 or newer, while the currently active rustc version is 1.62.0-dev
I cannot upgrade to anchor version 0.28.0 since it is breaking for switchboard-v2 crate.
@KirillLykov here is a script to reproduce the issue:
#!/usr/bin/env bash
set -e
# Set solana version
solana-install init 1.16.0
# Create a temp directory
mkdir data-too-large-error
pushd data-too-large-error
# Clone and build token-metadata program
git clone https://github.com/metaplex-foundation/metaplex.git
pushd metaplex
git checkout cbb590d5
pushd rust/token-metadata/program
cargo build-bpf
popd
popd
# Create an Anchor project(only for testing)
anchor init reproduce
cd reproduce
# Add the program to genesis
echo '[[test.genesis]]
address = "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s"
program = "../metaplex/rust/target/deploy/metaplex_token_metadata.so"' >> Anchor.toml
# Setup tests
yarn add @metaplex/js@=4.4.1
echo 'import * as anchor from "@coral-xyz/anchor";
import { Program } from "@coral-xyz/anchor";
import { programs } from "@metaplex/js";
import { Token, TOKEN_PROGRAM_ID } from "@solana/spl-token";
import { PublicKey } from "@solana/web3.js";
import { Reproduce } from "../target/types/reproduce";
const TOKEN_METADATA_PROGRAM_ID = new PublicKey(
"metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s"
);
const { CreateMetadata, MetadataDataData } = programs.metadata;
describe("reproduce", () => {
anchor.setProvider(anchor.AnchorProvider.env());
const program = anchor.workspace.Reproduce as Program<Reproduce>;
const payer = program.provider.publicKey;
it("reproduces", async () => {
const nftMintClient = await Token.createMint(
program.provider.connection,
// @ts-ignore
program.provider.wallet.payer,
payer,
null,
6,
TOKEN_PROGRAM_ID
);
// Create the metadata
const [metadata] = await PublicKey.findProgramAddress(
[
Buffer.from("metadata"),
TOKEN_METADATA_PROGRAM_ID.toBuffer(),
nftMintClient.publicKey.toBuffer(),
],
TOKEN_METADATA_PROGRAM_ID
);
const tx = new CreateMetadata(
{ feePayer: payer },
{
metadata,
metadataData: new MetadataDataData({
name: "test-nft",
symbol: "TEST",
uri: "https://nothing.com",
sellerFeeBasisPoints: 1,
creators: null,
}),
updateAuthority: payer,
mint: nftMintClient.publicKey,
mintAuthority: payer,
}
);
try {
await program.provider.sendAndConfirm(tx);
} catch (e) {
console.log(e);
return;
}
throw new Error("Did not reproduce");
});
});
' > tests/reproduce.ts
# Run tests
anchor test
# Return back to the initial directory
popd
The script uses Anchor only to test from TypeScript and the program that returns this error is not an Anchor program.
1.14.18
CLI solves the problemSeems like there is something wrong with the program binary. Running the tests with 1.16.0
works if the program was compiled with 1.14.18
.
solana-install init 1.14.18
pushd data-too-large-error/metaplex/rust/token-metadata/program
cargo build-bpf
popd
pushd data-too-large-error/reproduce
solana-install-init 1.16.0
anchor test
popd
Reproduced the problem with script above, note that works only with anchor 0.28.0
Based on the following runs, looks like compiler toolchain (not sdk) is the source of problem:
# OK
RUST_LOG=debug /home/klykov/.local/share/solana/install/releases/1.14.18/solana-release/bin/cargo-build-bpf --sbf-sdk "/home/klykov/.local/share/solana/install/releases/1.14.18/solana-release/bin/sdk/bpf"
# OK
RUST_LOG=debug /home/klykov/.local/share/solana/install/releases/1.14.18/solana-release/bin/cargo-build-bpf --sbf-sdk "/home/klykov/.local/share/solana/install/releases/1.16.0/solana-release/bin/sdk/sbf"
# Fails
RUST_LOG=debug /home/klykov/.local/share/solana/install/releases/1.16.0/solana-release/bin/cargo-build-bpf --sbf-sdk "/home/klykov/.local/share/solana/install/releases/1.16.0/solana-release/bin/sdk/sbf"
# Fails
RUST_LOG=debug /home/klykov/.local/share/solana/install/releases/1.16.0/solana-release/bin/cargo-build-bpf --sbf-sdk "/home/klykov/.local/share/solana/install/releases/1.14.18/solana-release/bin/sdk/bpf"
@dmakarov might be able to assist if this indeed is a toolchain issue
I was facing the same problem here, I fixed it by downgrading Solana Tool Suite from 1.16.1
to 1.14.18
without additional changes.
I get the same problem with GPL, tests work with1.14.18
but fail with 1.16.1
I get the same problem with GPL, tests work with
1.14.18
but fail with1.16.1
How to run GPL tests? Is running cargo test
sufficient to reproduce the problem?
How to run GPL tests? Is running
cargo test
sufficient to reproduce the problem? Yes,cargo test-sbf
for the integrations tests
@acheroncrypto From discussion with @dmakarov, looks like the problem is with the inconsistency of the dependencies version used by metaplex program, solana installation and anchor.
I've created a branch https://github.com/KirillLykov/metaplex-program-library/tree/klykov/sol-issue-31960 based on the commit cbb590d5
provided in the script above. This branch updates solana-sdk to 1.16.2 so it is aligned with solana installation, it updates anchor to 0.28 and updates borsh. This resolves the problem.
@KirillLykov thanks but I'll note that those versions hasn't changed in ~2 years and they were also tested with every Solana release within the last ~2 years(without having to change versions).
This will hurt composability quite a bit because not only you'd have to upgrade your own dependencies but you'd also potentially need to wait for all of the subdependencies to upgrade as well.
@KirillLykov thanks but I'll note that those versions hasn't changed in ~2 years and they were also tested with every Solana release within the last ~2 years(without having to change versions).
This will hurt composability quite a bit because not only you'd have to upgrade your own dependencies but you'd also potentially need to wait for all of the subdependencies to upgrade as well.
It looks like it is not on our side directly, we suspect that the problem is related to the borsh update (https://github.com/solana-labs/solana/pull/30975#issuecomment-1594253543)
In this particular case, it was enough to update only direct dependencies of MPL. It looks like the master of this repo already uses quite recent version of solana / anchor so updating is hopefully not a problem (if it is only about mpl) https://github.com/metaplex-foundation/metaplex-program-library/issues/1137
After downgrading to 1.14.20
, the version of cargo used in cargo-build-sbf
is 1.62-dev
. And a lot indirect dependencies (i.e. spl-token) need cargo version to be greater than 1.64
, how to solve this?
After downgrading to
1.14.20
, the version of cargo used incargo-build-sbf
is1.62-dev
. And a lot indirect dependencies (i.e. spl-token) need cargo version to be greater than1.64
, how to solve this?
is this an opensource project? I want to check the cargo tree to see what do you mean. I guess your project depends on some crates that are requiring 1.64
but you solana installation is on 1.62
, this should not be a problem I guess? Not sure
After downgrading to
1.14.20
, the version of cargo used incargo-build-sbf
is1.62-dev
. And a lot indirect dependencies (i.e. spl-token) need cargo version to be greater than1.64
, how to solve this?is this an opensource project? I want to check the cargo tree to see what do you mean. I guess your project depends on some crates that are requiring
1.64
but you solana installation is on1.62
, this should not be a problem I guess? Not sure
It's one of the most reported problems and the error message is not very helpful(https://github.com/solana-labs/solana/issues/31428).
Here are some examples from the StackExchange:
It's a miracle if you can compile a program with 1.14
CLI without a lock file(or downgrading all of the crates that bumped their MSRV past 1.62
).
After downgrading to
1.14.20
, the version of cargo used incargo-build-sbf
is1.62-dev
. And a lot indirect dependencies (i.e. spl-token) need cargo version to be greater than1.64
, how to solve this?is this an opensource project? I want to check the cargo tree to see what do you mean. I guess your project depends on some crates that are requiring
1.64
but you solana installation is on1.62
, this should not be a problem I guess? Not sureIt's one of the most reported problems and the error message is not very helpful(#31428).
Here are some examples from the StackExchange:
* https://solana.stackexchange.com/questions/6526/error-package-winnow-v0-4-4-cannot-be-built-because-it-requires-rustc-1-64-0 * https://solana.stackexchange.com/questions/7074/error-when-anchor-build-package-constant-time-eq-v0-3-0-cannot-be-built * https://solana.stackexchange.com/questions/6722/error-when-anchor-build-package-toml-datetime-v0-6-2-cannot-be-built
It's a miracle if you can compile a program with
1.14
CLI without a lock file(or downgrading all of the crates that bumped their MSRV past1.62
).
I was confused with the original message, I thought it is about solana and borsh update which lead to the current problem.
Now it looks to me that this is problem with cargo version https://github.com/solana-labs/solana/issues/31428 and, hence, is unrelated to the Invoked an instruction with data ...
problem, please correct me if I'm wrong
I was confused with the original message, I thought it is about solana and borsh update which lead to the current problem. Now it looks to me that this is problem with cargo version #31428 and, hence, is unrelated to the
Invoked an instruction with data ...
problem, please correct me if I'm wrong
The problem in #31428 is not directly related to the current issue but it's still related in a sense that when people upgrade to 1.16
, their program compiles but they get a runtime error with Invoked an instruction with data...
and when they try to downgrade to 1.14
(which solves the runtime problem) they get a compile error due to MSRV, hence the comment https://github.com/solana-labs/solana/issues/31960#issuecomment-1644450572.
@KirillLykov I finally have my problem solved by following #31775. I used to think the solution can either be upgrading to 1.16 or downgrading to 1.14.
However, for upgrading. mpl-token-metadata
still uses the old borsh, so it cannot be compiled with solana-program
.
So it doesn't work in my case.
For downgrading and using stable toolchain. Due to underlying libraries updated their dependencies, now they conflict with old cargo(1.62).
Upgrade of solana-cli
from 1.14 to 1.16 leads to problems with compiled on-chain program.
What we know:
Invoked an instruction with data that is too large (12884933012 > 10240)
My understanding is that it happens because the client program which triggers execution of the on-chain program serializes instruction data using borsh 0.9, while on-chain program uses newer version to deserialize it. (correct me if I'm wrong).
Do we know if borsh serialization was used to write serialized data to on-chain accounts? Like program with borsh 0.9 writes data in one format, while a program with borsh 0.10 reads this data and misinterprets it?
There is another issue explained in The error message for MSRV mentioned here. My understanding is that it happens when the user updated their installation to the latest but later returned back to 1.14 so that some crates use versions that cannot be compiled with the rustc version used internally by cargo-build-sbf. (Correct me if I'm wrong)
@acheroncrypto could you confirm that this summary is correct?
@acheroncrypto could you confirm that this summary is correct?
Yes, it's a good summary overall but I'm not fully convinced of the analysis for the following reasons.
- Problem: program compiled with toolchain 1.16 but it depends on previous versions of anchor 0.28 and solana crates. In this case, we get runtime error
Invoked an instruction with data that is too large (12884933012 > 10240)
Anchor version is irrelevant here, as I mentioned in my script post, Anchor is only being used for the client side tests and not as a program crate. Here are the dependencies that the program in the reproducable script example uses: https://github.com/metaplex-foundation/metaplex/blob/cbb590d53a3a9718ecf2b526580952fc6d10baaf/rust/token-metadata/program/Cargo.toml#L15-L23
- Most probably cause: borsh upgrade from 0.9 to 0.10. Introduced in Bump borsh to 0.10.3 #30975 (comment)
My understanding is that it happens because the client program which triggers execution of the on-chain program serializes instruction data using borsh 0.9, while on-chain program uses newer version to deserialize it. (correct me if I'm wrong).
Even if that's the case, why does the on-chain program use 0.10
? The program in the reproducable script has a lock file so the version of solana-program
and borsh
crates don't actually change when you change the toolchain/CLI version. Why would changing the build-tools/CLI version have an effect on borsh
de/serialization when the crate versions hasn't changed at all?
Do we know if borsh serialization was used to write serialized data to on-chain accounts? Like program with borsh 0.9 writes data in one format, while a program with borsh 0.10 reads this data and misinterprets it?
To my knowledge, borsh
serialization is still the same and the breaking change comes from a new required deserialization method in the API. How the data is serialized hasn't changed and it's not like we have a new borsh
that interprets the data differently after many years. That would totally break everything so it would most likely never happen(🤞).
Solana build tools is not opinionated on which de/serialization was used in the program, so if there is a program with a lock file, that used to work, should continue to work independent of the build tools/CLI version used. Why? Because the program doesn't even have a dependency to the new borsh
version. (https://github.com/metaplex-foundation/metaplex/blob/cbb590d53a3a9718ecf2b526580952fc6d10baaf/rust/Cargo.lock#L392-L435)
@acheroncrypto thanks for the comments, I somehow missed the part of why borsh has been upgraded after cli upgrade. Should be lock file which controls that, need to dig a bit more into that
I figured this out finally! I was able to reproduce this reliably by upgrading my tools to 1.16, and then building and running a program that uses the 1.14 crates.
The program just does a token transfer in a CPI: https://github.com/solana-labs/solana-program-library/tree/master/examples/rust/transfer-tokens
If you build the program with 1.16 tools, 1.14 crates, and dump the raw byte representation of the transfer instruction, ie:
let p = &ix as *const _ as *const u8;
let bytes = unsafe { core::slice::from_raw_parts(p, 30) };
msg!("{:?}", &bytes);
You'll get:
[2023-08-07T22:57:59.720273041Z DEBUG solana_runtime::message_processor::stable_log] Program log: [6, 221, 246, 225, 215, 101, 161, 147, 217, 203, 225, 70, 206, 235, 121, 172, 28, 180, 133, 237, 95, 91, 55, 145, 58, 140, 245, 133, 126, 255]
And with 1.14 tools, 1.14 crates, you get:
[2023-08-07T22:57:38.652318633Z DEBUG solana_runtime::message_processor::stable_log] Program log: [208, 124, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 88, 125, 0, 0, 3, 0]
What's going on here? This time, we can blame the Rust language! (But it's not their fault, because they explicitly say that they can change the internals)
The Solana tools v1.16 use a Rust compiler v1.68. In Rust v1.66, the byte layout of Vec
changed, so we introduced some new types that maintain the old byte layout of important types such as Vec
in https://github.com/solana-labs/solana/pull/30192
To preserve ABI compatibility, when you call invoke
or invoke_signed
, the 1.16 crates have an extra step where they transform from the normal Rust representation to the StableInstruction
variant, which maintains the old ABI for Vec
: https://github.com/solana-labs/solana/blob/b7c2ad5b67990161d9972d1fa6e81510a023a20d/sdk/program/src/program.rs#L296
But in our case, we're using the 1.16 compiler, which breaks Vec
ABI, with the 1.14 crates, which have no correction for Vec
, so things break!
Until we can backport those fixes to 1.14, the workaround for this issue is to be sure to always use the 1.16 crates with the 1.16 compiler, otherwise your program will break Vec
ABI and fail runtime checks.
This explanation makes sense, nice spot @joncinque!
Until we can backport those fixes to 1.14, the workaround for this issue is to be sure to always use the 1.16 crates with the 1.16 compiler, otherwise your program will break
Vec
ABI and fail runtime checks.
The issue has been:
1.14
CLI -> MSRV error1.16
CLI with previous crates -> Data too large error1.16
CLI and crates -> Not all crates work with the latest solana-program
due to the breaking changes(partly resolved in https://github.com/solana-labs/solana/pull/32511)It was a perfect storm for dependency issues. Glad it's being resolved now!
@acheroncrypto can we close this issue if it looks resolved?
@acheroncrypto can we close this issue if it looks resolved?
The root cause of the issue has been identified but many people are still blocked with an older solana-program
version. Probably best to close after the fix is backported to 1.14
(https://github.com/solana-labs/solana/pull/30192#issuecomment-1668715322).
After a discussion with the team offline, we've decided that the change is too risky to backport. Releases are coordinated with a certain version of the Rust compiler, and it is dangerous to encourage mixing crate and Rust versions in any capacity.
It's a bigger risk for programs to behave unexpectedly or incorrectly, so even though it's painful in the short term, people must find a way to get their dependencies up to date to use the newer compiler.
If you still want to use old versions and get errors from dependencies that are automatically updated, I encourage you to use an old working lockfile. https://github.com/solana-labs/solana-program-library/tree/ca7d3876313ed3ea6138aa3c1e00c23896e91949 is the last version of SPL that works with 1.14. To properly run tests, I used this old version, where all the dependencies are properly pinned to avoid pulling in newer crates.
And while I don't recommend it for real program deployments, you can always patch the Solana crates with a local copy that has https://github.com/solana-labs/solana/pull/32797 applied.
This issue is dangerous and should be highlighted in the core documentation. It can cause issues with redeployment of old, but extremely stable programs.
Based on the discussion here, my understanding is that the safest way to proceed is to use latest 1.16 version for all three: solana-cli, runtime, and solana-program. Am I right, @jon?
Based on the discussion here, my understanding is that the safest way to proceed is to use latest 1.16 version for all three: solana-cli, runtime, and solana-program. Am I right, @jon?
Yep, that's right -- at the very least, they need to use 1.16 for solana-program
and the CLI, and then maybe for any other test dependencies, like the runtime
Got this error again:
Program returned error: Instruction passed to inner instruction is too large (18446744073709551615 > 1280) solana-cli 1.17.14
Problem
Some programs raise
Invoked an instruction with data that is too large
error with1.16.0
release.I wasn't able to reproduce this error in a smaller program but I can share 3 different programs where this happens.
Logs from this instruction with
1.16.0
(platform-tools v1.37):Pre
1.16.0
(platform-tools v1.29):All examples work pre
1.16.0
and don't work with1.16.0
and seems like others who have used the1.16.0
version have run into this issue as well. See https://github.com/solana-labs/solana/issues/31337#issue-1682750647, https://github.com/solana-labs/solana/issues/30271#issuecomment-1518740240.