Noble BLS fails BLS tests on Prague, while MCL does not

jochem-brouwer commented 3 weeks ago

When working on t8n, we failed to build fixtures for BLS (EIP-2537). This is because our EVM defaults to NobleBLS, but, in the blockchain/state test runner we:

  let bls: EVMBLSInterface
  if (argv.bls !== undefined && argv.bls.toLowerCase() === 'noble') {
    console.log('BLS library used: Noble (JavaScript)')
    bls = new NobleBLS()
  } else {
    await mcl.init(mcl.BLS12_381)
    bls = new MCLBLS(mcl)
    console.log('BLS library used: MCL (WASM)')
  }

default to MCL. Using MCL, the tests are generated.

The fixtures below have been generated by geth (I have done this locally) and are included as zip in this issue. To run tests, unzip the fixtures inside ethereum-tests and now run blockchain/state tests: (from vm package)

npm run test:blockchain -- --fork=Prague --dir=../fixtures/blockchain_tests (pass)
npm run test:blockchain -- --fork=Prague --dir=../fixtures/blockchain_tests --bls=noble (fail)
npm run test:state -- --fork=Prague --dir=../fixtures/state_tests (pass)
npm run test:state -- --fork=Prague --dir=../fixtures/state_tests --bls=noble (fail)

TODOs:

[ ] Figure out why NobleBLS fails
[ ] Add a specific state/blockchain test which will also test the other dependency (so not the default) to the CI action. This should only run the 2537 tests, no need to run all other tests
[ ] Also, the choice to use MCL is likely made in the CI for the performance benefit, but we should likely switch to the default here in case no BLS is passed?

fixtures.zip

holgerd77 commented 3 weeks ago

No, we should never make MCL default again, and rather work our heads off to get Noble passing. 😂 A WASM free VM is one of the core value propositions of this release round (security, especially in the light of 7702,).

holgerd77 commented 3 weeks ago

(in doubt ask @paulmillr )

paulmillr commented 3 weeks ago

I will dive into that.

To ease the ramp up, could you please share a specific code sample & vector that fails?

jochem-brouwer commented 2 weeks ago

Hi @paulmillr, I wanted to ping you once I get deeper into this, but did not have the bandwidth. However, if you want, you can see the current failing test vectors using: (you need to extract the fixtures.zip inside ./packages/ethereum-tests before running, the fixtures are in the OP)

npm run test:state -- --fork=Prague --dir=../fixtures/state_tests --bls=noble (59 fail)

npm run test:blockchain -- --fork=Prague --dir=../fixtures/blockchain_tests --bls=noble (118 fail, but this is 59*2, since each blockchain test will throw 2 errors if it fails (in most cases))

Note: you can select a single test with --test, e.g: npm run test:state -- --fork=Prague --dir=../fixtures/state_tests --bls=noble --test=tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--bls_g1add_point_not_on_curve-]

Here is the output for the state tests (only the errors)

Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--bls_g1add_point_not_on_curve-]:
    [ 0.02 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--invalid_point_a_1-]:
    [ 0.018 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--invalid_point_a_2-]:
    [ 0.016 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--invalid_point_a_3-]:
    [ 0.015 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--invalid_point_a_4-]:
    [ 0.017 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--invalid_point_a_5-]:
    [ 0.016 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--invalid_point_b_1-]:
    [ 0.014 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--invalid_point_b_2-]:
    [ 0.015 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--invalid_point_b_3-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--invalid_point_b_4-]:
    [ 0.016 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1add.py::test_invalid[fork_Prague-state_test--invalid_point_b_5-]:
    [ 0.014 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1msm.py::test_invalid[fork_Prague-state_test---bls_g1multiexp_point_not_on_curve-]:
    [ 0.031 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1msm.py::test_invalid[fork_Prague-state_test---bls_g1multiexp_g1_not_in_correct_subgroup-]:
    [ 0.029 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1msm.py::test_invalid[fork_Prague-state_test---invalid_point_1-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1msm.py::test_invalid[fork_Prague-state_test---invalid_point_2-]:
    [ 0.014 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1msm.py::test_invalid[fork_Prague-state_test---invalid_point_3-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1msm.py::test_invalid[fork_Prague-state_test---invalid_point_4-]:
    [ 0.014 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1msm/valid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1msm.py::test_valid[fork_Prague-state_test-max_discount-]:
    [ 0.018 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1msm/valid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1msm.py::test_valid[fork_Prague-state_test-max_discount_plus_1-]:
    [ 0.019 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1mul.py::test_invalid[fork_Prague-state_test--bls_g1mul_point_not_on_curve-]:
    [ 0.02 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1mul.py::test_invalid[fork_Prague-state_test--bls_g1mul_g1_not_in_correct_subgroup-]:
    [ 0.019 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1mul.py::test_invalid[fork_Prague-state_test--invalid_point_1-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1mul.py::test_invalid[fork_Prague-state_test--invalid_point_2-]:
    [ 0.012 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1mul.py::test_invalid[fork_Prague-state_test--invalid_point_3-]:
    [ 0.012 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1mul.py::test_invalid[fork_Prague-state_test--invalid_point_4-]:
    [ 0.012 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1mul.py::test_invalid[fork_Prague-state_test--bls_g1mul_not_in_subgroup_1-]:
    [ 0.02 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1mul.py::test_invalid[fork_Prague-state_test--bls_g1mul_not_in_subgroup_2-]:
    [ 0.025 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g1mul/valid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1mul.py::test_valid[fork_Prague-state_test-bls_g1mul_(q*P1)-]:
    [ 0.012 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--bls_g2add_point_not_on_curve-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--invalid_point_a_1-]:
    [ 0.014 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--invalid_point_a_2-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--invalid_point_a_3-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--invalid_point_a_4-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--invalid_point_a_5-]:
    [ 0.017 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--invalid_point_b_1-]:
    [ 0.012 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--invalid_point_b_2-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--invalid_point_b_3-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--invalid_point_b_4-]:
    [ 0.015 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2add/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2add.py::test_invalid[fork_Prague-state_test--invalid_point_b_5-]:
    [ 0.018 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2msm.py::test_invalid[fork_Prague-state_test--bls_g2multiexp_point_not_on_curve-]:
    [ 0.064 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2msm.py::test_invalid[fork_Prague-state_test--bls_pairing_g2_not_in_correct_subgroup-]:
    [ 0.065 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2msm.py::test_invalid[fork_Prague-state_test--invalid_point_a_1-]:
    [ 0.011 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2msm.py::test_invalid[fork_Prague-state_test--invalid_point_a_2-]:
    [ 0.012 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2msm.py::test_invalid[fork_Prague-state_test--invalid_point_a_3-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2msm.py::test_invalid[fork_Prague-state_test--invalid_point_a_4-]:
    [ 0.012 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2msm/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2msm.py::test_invalid[fork_Prague-state_test--bls_g2mul_not_in_subgroup-]:
    [ 0.038 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2mul.py::test_invalid[fork_Prague-state_test--bls_g2mul_point_not_on_curve-]:
    [ 0.037 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2mul.py::test_invalid[fork_Prague-state_test--bls_g2mul_g2_not_in_correct_subgroup-]:
    [ 0.039 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2mul.py::test_invalid[fork_Prague-state_test--invalid_point_a_1-]:
    [ 0.02 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2mul.py::test_invalid[fork_Prague-state_test--invalid_point_a_2-]:
    [ 0.013 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2mul.py::test_invalid[fork_Prague-state_test--invalid_point_a_3-]:
    [ 0.014 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2mul.py::test_invalid[fork_Prague-state_test--invalid_point_a_4-]:
    [ 0.014 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2mul.py::test_invalid[fork_Prague-state_test--bls_g2mul_not_in_subgroup-]:
    [ 0.038 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2mul/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2mul.py::test_invalid[fork_Prague-state_test--bls_g2mul_not_in_subgroup_times_2-]:
    [ 0.043 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2mul/valid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2mul.py::test_valid[fork_Prague-state_test-bls_g2mul_(q*P2)-]:
    [ 0.018 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_g2mul/valid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g2mul.py::test_valid[fork_Prague-state_test-bls_g2mul_(q*G2)-]:
    [ 0.02 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_pairing/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_pairing.py::test_invalid[fork_Prague-state_test--multi_inf_plus_g1_P_g2_inf_1-]:
    [ 0.118 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_pairing/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_pairing.py::test_invalid[fork_Prague-state_test--P1_not_in_subgroup-]:
    [ 0.011 secs ] the state roots should match (successful tx run)
Errors thrown in file: prague/eip2537_bls_12_381_precompiles/bls12_pairing/invalid.json test: tests/prague/eip2537_bls_12_381_precompiles/test_bls12_pairing.py::test_invalid[fork_Prague-state_test--P2_not_in_subgroup-]:
    [ 0.012 secs ] the state roots should match (successful tx run)

@holgerd77 Ah great point, in that case we should definitely point our test runner to use Noble by default!

holgerd77 commented 2 weeks ago

Will also have a look into this in the next couple of days, maybe I can spot relatively easy what's wrong after having done the integration!

paulmillr commented 1 week ago

The library itself had no errors - just its usage was not correct.

paulmillr commented 1 week ago

Also multiply can be replaced with multiplyUnsafe which will speed up everything massively. unsafe should be ok, assuming the data is non-private / public (EVM?)

jochem-brouwer commented 1 week ago

The EVM should only be ran in public contexts, so all input data to evm (i.e. tx calldata, contract code, etc.) is publicly known.

ethereumjs / ethereumjs-monorepo

Noble BLS fails BLS tests on Prague, while MCL does not #3609