BUG: Failed to parse root metadata: unknown variant `ecdsa`, expected one of `rsa`, `ed25519`, `ecdsa-sha2-nistp256`

[astoycos]

Description

This error randomly started popping up today in our CI and I was able to reproduce locally , I tried bumping our sigstore-rs dep up to main to see if that fixed the issue with no luck, a push in the right direction to fix this would be much appreciated and I'm happy to help out :)

It occurs when pre-fetching tuf data like so

        spawn_blocking(|| {
            let tuf =
                sigstore::trust::sigstore::SigstoreTrustRoot::new(get_tuf_path().as_deref()).map_err(|e| {
                    anyhow!(
                        "Error spawning blocking task to build sigstore repo inside of tokio: {}",
                        e
                    )
                })?;

            tuf.prefetch().map_err(|e| {
                anyhow!(
                    "Error spawning blocking task to prefetch tuf data inside of tokio: {}",
                    e
                )
            })
        })

OS INFO: Fedora 38

[astoycos@nfvsdn-03 bpfman]$ uname -r 
6.4.11-200.fc38.x86_64

ERROR:

[INFO  bpfman::oci_utils::cosign] Starting Cosign Verifier, downloading data from Sigstore TUF repository
thread 'main' panicked at /home/astoycos/go/src/github.com/bpfman/bpfman/bpfman/src/lib.rs:155:22:
failed to initialize image manager: Error spawning blocking task to build sigstore repo inside of tokio: Failed to parse root metadata: unknown variant `ecdsa`, expected one of `rsa`, `ed25519`, `ecdsa-sha2-nistp256` at line 9 column 22

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.81/src/error.rs:83:36
   1: bpfman::oci_utils::cosign::fetch_sigstore_tuf_data::{{closure}}::{{closure}}::{{closure}}
             at ./bpfman/src/oci_utils/cosign.rs:115:21
   2: core::result::Result<T,E>::map_err
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/result.rs:829:27
   3: bpfman::oci_utils::cosign::fetch_sigstore_tuf_data::{{closure}}::{{closure}}
             at ./bpfman/src/oci_utils/cosign.rs:114:17
   4: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/task.rs:42:21
   5: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/task/core.rs:328:17
   6: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/loom/std/unsafe_cell.rs:16:9
   7: tokio::runtime::task::core::Core<T,S>::poll
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/task/core.rs:317:13
   8: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/task/harness.rs:485:19
   9: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/panic/unwind_safe.rs:272:9
  10: std::panicking::try::do_call
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:552:40
  11: __rust_try
  12: std::panicking::try
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:516:19
  13: std::panic::catch_unwind
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panic.rs:146:14
  14: tokio::runtime::task::harness::poll_future
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/task/harness.rs:473:18
  15: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/task/harness.rs:208:27
  16: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/task/harness.rs:153:15
  17: tokio::runtime::task::raw::poll
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/task/raw.rs:271:5
  18: tokio::runtime::task::raw::RawTask::poll
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/task/raw.rs:201:18
  19: tokio::runtime::task::UnownedTask<S>::run
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/task/mod.rs:453:9
  20: tokio::runtime::blocking::pool::Task::run
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/pool.rs:159:9
  21: tokio::runtime::blocking::pool::Inner::run
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/pool.rs:513:17
  22: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/pool.rs:471:13
  23: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/sys_common/backtrace.rs:155:18
  24: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/thread/mod.rs:528:17
  25: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/panic/unwind_safe.rs:272:9
  26: std::panicking::try::do_call
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:552:40
  27: __rust_try
  28: std::panicking::try
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:516:19
  29: std::panic::catch_unwind
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panic.rs:146:14
  30: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/thread/mod.rs:527:30
  31: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/ops/function.rs:250:5
  32: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/alloc/src/boxed.rs:2020:9
  33: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/alloc/src/boxed.rs:2020:9
  34: std::sys::pal::unix::thread::Thread::new::thread_start
             at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/sys/pal/unix/thread.rs:108:17
  35: start_thread
  36: __clone3
stack backtrace:
   0:     0x558fa6510812 - std::backtrace_rs::backtrace::libunwind::trace::h62f3b47628dd17b6
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
   1:     0x558fa6510812 - std::backtrace_rs::backtrace::trace_unsynchronized::h3892e36781ebd7a9
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x558fa6510812 - std::sys_common::backtrace::_print_fmt::h5753abb8ee14651e
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x558fa6510812 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h641e95e38120aa19
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x558fa653efbc - core::fmt::rt::Argument::fmt::h39826a56c345298d
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/fmt/rt.rs:142:9
   5:     0x558fa653efbc - core::fmt::write::hcfbdb7435841633f
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/fmt/mod.rs:1153:17
   6:     0x558fa650cd4f - std::io::Write::write_fmt::hcceb62a2bad26450
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/io/mod.rs:1843:15
   7:     0x558fa65105e4 - std::sys_common::backtrace::_print::ha68066140c1e1d27
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x558fa65105e4 - std::sys_common::backtrace::print::h48d14ab80b242509
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x558fa6511c0b - std::panicking::default_hook::{{closure}}::h6c98b71ccc05382a
  10:     0x558fa6511963 - std::panicking::default_hook::h78c2716692948d24
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:292:9
  11:     0x558fa65120ad - std::panicking::rust_panic_with_hook::hc8ea6fc0780cadf2
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:779:13
  12:     0x558fa6511f82 - std::panicking::begin_panic_handler::{{closure}}::hbadcf855f0ac336d
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:657:13
  13:     0x558fa6510ce6 - std::sys_common::backtrace::__rust_end_short_backtrace::hd709bf062ed2b8bd
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/sys_common/backtrace.rs:171:18
  14:     0x558fa6511cb4 - rust_begin_unwind
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:645:5
  15:     0x558fa4a6b1c5 - core::panicking::panic_fmt::hc88aa6bdd27ff112
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/panicking.rs:72:14
  16:     0x558fa4a6b7b3 - core::result::unwrap_failed::h2f69d9c547d2d005
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/result.rs:1654:5
  17:     0x558fa4a96b6f - core::result::Result<T,E>::expect::hcfbd6a870b3f5a1b
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/result.rs:1034:23
  18:     0x558fa4aec6a2 - bpfman::BpfManager::init_image_manager::{{closure}}::h3c7e3ddbd4f0562c
                               at /home/astoycos/go/src/github.com/bpfman/bpfman/bpfman/src/lib.rs:153:17
  19:     0x558fa4ae8ebf - bpfman::BpfManager::add_program::{{closure}}::h19ca0a6cff4e44fb
                               at /home/astoycos/go/src/github.com/bpfman/bpfman/bpfman/src/lib.rs:377:35
  20:     0x558fa4a7a333 - bpfman::load::execute_load_image::{{closure}}::h557f2a8b7ebc53b5
                               at /home/astoycos/go/src/github.com/bpfman/bpfman/bpfman/src/bin/cli/load.rs:80:10
  21:     0x558fa4a78a60 - bpfman::load::<impl bpfman::args::LoadSubcommand>::execute::{{closure}}::h02a8d3af5651e8e6
                               at /home/astoycos/go/src/github.com/bpfman/bpfman/bpfman/src/bin/cli/load.rs:24:76
  22:     0x558fa4b1a5a5 - bpfman::<impl bpfman::args::Commands>::execute::{{closure}}::h8bfa7e88f66106ef
                               at /home/astoycos/go/src/github.com/bpfman/bpfman/bpfman/src/bin/cli/main.rs:38:62
  23:     0x558fa4b1af02 - bpfman::main::{{closure}}::h72b11e5aabdf296f
                               at /home/astoycos/go/src/github.com/bpfman/bpfman/bpfman/src/bin/cli/main.rs:28:27
  24:     0x558fa4a837c4 - tokio::runtime::park::CachedParkThread::block_on::{{closure}}::h9b70e93d05561ecd
                               at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/park.rs:281:63
  25:     0x558fa4a835bb - tokio::runtime::coop::with_budget::h06a66e636ad7e640
                               at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/coop.rs:107:5
  26:     0x558fa4a835bb - tokio::runtime::coop::budget::h0bbc6a963156c379
                               at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/coop.rs:73:5
  27:     0x558fa4a835bb - tokio::runtime::park::CachedParkThread::block_on::hb2599f4a1c8584d7
                               at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/park.rs:281:31
  28:     0x558fa4b041a4 - tokio::runtime::context::blocking::BlockingRegionGuard::block_on::hf58b699ff77a1c0e
                               at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/blocking.rs:66:9
  29:     0x558fa4b033e5 - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}::hf27fd6ead1863423
                               at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  30:     0x558fa4b0e5f3 - tokio::runtime::context::runtime::enter_runtime::h68761dee0a4bdf8b
                               at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/runtime.rs:65:16
  31:     0x558fa4b03390 - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::h69850e37016e74e0
                               at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  32:     0x558fa4aa54ab - tokio::runtime::runtime::Runtime::block_on::hf46428fadcccf866
                               at /home/astoycos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/runtime.rs:350:45
  33:     0x558fa4a76440 - bpfman::main::h5867817d7bfa9594
                               at /home/astoycos/go/src/github.com/bpfman/bpfman/bpfman/src/bin/cli/main.rs:28:5
  34:     0x558fa4aaeaab - core::ops::function::FnOnce::call_once::h76242f295bd70996
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/ops/function.rs:250:5
  35:     0x558fa4b19f0e - std::sys_common::backtrace::__rust_begin_short_backtrace::h02ce1870b1a3ac83
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/sys_common/backtrace.rs:155:18
  36:     0x558fa4aa4241 - std::rt::lang_start::{{closure}}::hba297143c71f0b1f
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/rt.rs:166:18
  37:     0x558fa65054d3 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hdfd670beedd02cf9
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/core/src/ops/function.rs:284:13
  38:     0x558fa65054d3 - std::panicking::try::do_call::h1c46aa5e98fa3311
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:552:40
  39:     0x558fa65054d3 - std::panicking::try::hf7c9cd91394edf65
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:516:19
  40:     0x558fa65054d3 - std::panic::catch_unwind::he91c20b84a706ef3
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panic.rs:146:14
  41:     0x558fa65054d3 - std::rt::lang_start_internal::{{closure}}::hcc8f51788575684d
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/rt.rs:148:48
  42:     0x558fa65054d3 - std::panicking::try::do_call::hf6cddf9eb5d66920
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:552:40
  43:     0x558fa65054d3 - std::panicking::try::h78e2163eeecfd440
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panicking.rs:516:19
  44:     0x558fa65054d3 - std::panic::catch_unwind::h393e3e1b2f32a694
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/panic.rs:146:14
  45:     0x558fa65054d3 - std::rt::lang_start_internal::ha868a0cddc810206
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/rt.rs:148:20
  46:     0x558fa4aa421a - std::rt::lang_start::h8b2af377944a6473
                               at /rustc/a165f1f65015b1bd4afd2ec50700aaacf2e0c485/library/std/src/rt.rs:165:17
  47:     0x558fa4a764ee - main
  48:     0x7fe7fa368b8a - __libc_start_call_main
  49:     0x7fe7fa368c4b - __libc_start_main_alias_2
  50:     0x558fa4a6b965 - _start
  51:                0x0 - <unknown>

Version

using sigstore-rs:0.8.0 and sigstore-rs:main

[astoycos]

After some digging it seems to be an error originating in https://github.dev/awslabs/tough possibly?

[kommendorkapten]

Yes, I patched that last week and is waiting for it to be reviewed: https://github.com/awslabs/tough/pull/755

[astoycos]

Thanks @kommendorkapten Much appreciated

[astoycos]

@kommendorkapten Out of curiosity... we're using the 0.8.0 tag which should have had all the dep version's fixed, how did this manage to break us?

[kommendorkapten]

@astoycos the Sigstore TUF root was updated yesterday, where the keytype attribute changed to ecdsa from ecdsa-sha2-nistp256, and as awslabs/tough does not yet support that key type, it could not verify the updated TUF root. I think that's the answer to your question?

[kommendorkapten]

And this will continue to fail until awslabs/tough is updated, hope that my PR is accepted so this can be resolved for sigstore-rs.

[astoycos]

That makes sense, thanks for explaining :)

[flavio]

This is now addressed with the 0.9.0 release, which is already available on crates.io :partying_face: