Issue on moderately large programs

[delehef]

Hello,

I'm currently trying to run a 435 KiB ELF (elf.zip), and run into several issues.

1. with #[jolt::provable], which raises:

   thread 'main' panicked at [...]/jolt/tracer/src/emulator/memory.rs:31:19:
   index out of bounds: the len is 16777216 but the index is 147730206

2. increasing stack size with #[jolt::provable(stack_size = 16_000_000, memory_size = 100_000_000)] solves the first issue, but then raises:

   thread 'main' panicked at [...]/jolt/tracer/src/emulator/mmu.rs:532:22:
   Unknown memory mapping 53380D13.
   stack backtrace:
   [panicking, formatting & unwinding stuff]
   16:     0x55cd45cd971b - tracer::emulator::mmu::Mmu::trace_load::h61beb4fb67680baa
                              at [...]/jolt/tracer/src/emulator/mmu.rs:532:22
   17:     0x55cd45cd6fad - tracer::emulator::mmu::Mmu::load::h5da8d0212cc92ade
                              at [...]/jolt/tracer/src/emulator/mmu.rs:297:9
   18:     0x55cd45cd6fad - tracer::emulator::cpu::INSTRUCTIONS::{{closure}}::h79a65aba082cf0d7
                              at [...]/jolt/tracer/src/emulator/cpu.rs:2976:33
   19:     0x55cd45cd6fad - core::ops::function::FnOnce::call_once::ha734899cdfc763ad
                              at /rustc/b3aa8e7168a3d940122db3561289ffbf3f587262/library/core/src/ops/function.rs:250:5
   20:     0x55cd45cca664 - tracer::emulator::cpu::Cpu::tick_operate::hb1dd42bd6f103a4d
                              at [...]/jolt/tracer/src/emulator/cpu.rs:339:30
   21:     0x55cd45cc4379 - tracer::emulator::cpu::Cpu::tick::hff6e60e70aec136d
                              at [...]/jolt/tracer/src/emulator/cpu.rs:292:15
   22:     0x55cd45cc4379 - tracer::emulator::Emulator::tick::h5fadc54ac06bec4b
                              at [...]/jolt/tracer/src/emulator/mod.rs:133:9
   23:     0x55cd45cc4379 - tracer::trace::ha8898450a628ba43
                              at [...]/jolt/tracer/src/lib.rs:44:9
   24:     0x55cd45c5093c - jolt_core::host::Program::trace_analyze::h463dbf87fe61a022
                              at [...]/jolt/jolt-core/src/host/mod.rs:182:30
   25:     0x55cd45c5093c - guest::analyze_tree_hash::h826fb392cdf65f16
                              at [...]/jolt-tree/guest/src/lib.rs:6:1
   26:     0x55cd45c5093c - tree_store::main::h7b87ede3ebc1220f
                              at [...]/jolt-tree/src/main.rs:3:27
   27:     0x55cd45c4fa63 - core::ops::function::FnOnce::call_once::hd31a2da5d6d4aac2
                              at /rustc/b3aa8e7168a3d940122db3561289ffbf3f587262/library/core/src/ops/function.rs:250:5
   28:     0x55cd45c4fa63 - std::sys_common::backtrace::__rust_begin_short_backtrace::h25f4ca19dbe3b754
                              at /rustc/b3aa8e7168a3d940122db3561289ffbf3f587262/library/std/src/sys_common/backtrace.rs:154:18
   29:     0x55cd45c53f75 - main
   30:     0x7f89bfd86fce - __libc_start_call_main
   31:     0x7f89bfd87089 - __libc_start_main_impl
   32:     0x55cd45c4f995 - _start
   33:                0x0 - <unknown>

Any idea of what may be going wrong?

I advise to test it with my fork of jolt which streams traces to disk, as they grow quite large (>32GiB).

[ncitron]

Can you send me a repo that I can use that has you jolt program in it rather than just an elf file so I can better debug?

[delehef]

Sure thing, here is a repo following the default template.

[ncitron]

Ok so it looks like trivial_tree even when running without the std feature is trying to do something that requires the std (or some core thing which we don't yet support) which causes it to null pointer when it tries to do some display stuff. I was able to fix it by enabling our experimental standard library support which you can do by adding the guest-std feature to your jolt import in the guest.

Also note we've renamed the std feature which you have to enable when importing jolt from the host to host so you may need to change this if you update jolt.

Unfortunately we're hunting down a bug in jolt right now which breaks the std support sometimes when proving so it still errors when proving, but it now traces successfully. Will keep you updated here on that bug.

[delehef]

Thanks for the detailed report!

which causes it to null pointer when it tries to do some display stuff

There is no MMU, so the symptoms are the PC jumping to 0 and Jolt starting executing from there, right?

Also note we've renamed the std feature which you have to enable when importing jolt from the host to host

I believe I did, but probably pushed after you cloned my repo; sorry for that.

[ncitron]

Yeah what we think is happening is its trying to load a function related to either a formatter or a error handling thing and it doesn't exist so its jumping to zero.

We are trying to chase down issues with the standard library and harden it a bit more, and eventually making it the default so these issues should crop up less.

[delehef]

Interesting, out of pure curiosity, I'd love to hear about the root issue when you will have narrowed it down.

Good hunt!

[delehef]

Hey, the same program fails again on the latest master with:

emulator/cpu.rs:337:45:
called `Option::unwrap()` on a `None` value

Could you please take a look?

[ncitron]

I was just able to generate a proof of your program successfully. I used stack_size = 200_0000 and memory_size = 100_000_000.

I just pushed a new release to our rust toolchain this morning though so that may need to be installed. To ensure you have this, reinstall the jolt cli (since we only added the installer for our new toolchain recently) by running cargo +nightly install --git https://github.com/a16z/jolt --force --bins jolt and install the toolchain using jolt install-toolchain.

[delehef]

It looks like I still have an issue w.r.t. dependencies:

[...]
  * `lib` target `guest`
  * `bin` target `guest`
   Compiling jolt-sdk v0.1.0 (https://github.com/a16z/jolt#8852dd14)
error[E0463]: can't find crate for `postcard`
 --> [...]/.cargo/git/checkouts/jolt-6b856340b98daf0c/8852dd1/jolt-sdk/src/lib.rs:6:9
  |
6 | pub use postcard;
  |         ^^^^^^^^ can't find crate

This is being built on latest master (8852dd14).

Edit: this is avoided in disabling the guest-std feature, but this should probably be fixed so that all combinations of features compile.

Thanks for the help!