Build failure on FreeBSD

[nkosi23]

Describe the bug When building the full service API on FreeBSD, everything builds smoothly until the 798th package out of 924, the build error is:

To Reproduce Steps to reproduce the behavior:

Install rustup on FreeBSD as explained in: https://gist.github.com/fabiotatsuo/e88b025baf3dfa47683419948723706f

Install the build dependencies with the command:

pkg install cmake protobuf llvm llvm-devel sqlite3 sqlcipher git bash

Expected behavior The build completes without error. Rusts binaries are usually easily portable to FreeBSD, it looks like this issue could just be a matter of adding a few config files to support FreeBSD.

Environment (please complete the following information):

• OS: FreeBSD
• Version 13

Logs

error: failed to run custom build command for `hidapi v2.4.1`

Caused by:
  process didn't exit successfully: `/home/mobilecoin/full-service/target/release/build/hidapi-0f1b6d5217db2a16/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=HIDAPI_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=HIDAPI_STATIC
  cargo:rerun-if-env-changed=HIDAPI_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=HIDAPI_STATIC
  cargo:rerun-if-env-changed=HIDAPI_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hidapi-2.4.1/build.rs:131:41:
  Unable to find hidapi: Could not run `PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 pkg-config --libs --cflags hidapi`
  The pkg-config command could not be found.

  Most likely, you need to install a pkg-config package for your OS.
  Try `apt install pkg-config`, or `yum install pkg-config`,
  or `pkg install pkg-config`, or `apk add pkgconfig` depending on your distribution.

  If you've already installed it, ensure the pkg-config command is one of the
  directories in the PATH environment variable.

  If you did not expect this build to link to a pre-installed system library,
  then check documentation of the hidapi crate for an option to
  build the library from source, or disable features or dependencies
  that require pkg-config.
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And the build eventually ends with these logs.

Additional context Add any other context about the problem here.

[nick-mobilecoin]

if you want FreeBSD native you might be able to get by using https://github.com/mobilecoinofficial/full-service/blob/main/tools/build-fs.sh

./tools/build-fs.sh main

The error your getting is because the enclave css(not cascading style sheets) environment variables are not specified via:

• CONSENSUS_ENCLAVE_CSS
• INGETS_ENCLAVE_CSS

When these aren't specified then the build tries to build the enclaves to generate the css files. However the enclave builds are limited to x86_64 linux.

The build-fs.sh will download the current css files for you and then build full service using them.

you could also use the docker image. The mob command will invoke docker for the appropriate image.

./mob prompt
cargo build

[nkosi23]

@nick-mobilecoin Thanks a lot for your swift reply! However I was actually using the build-fs.sh script, here is the full output of the terminal session (but you helped me find out the actual issue):

root@mobilecoinfullservicetest:/home/mobilecoin/full-service/tools # ./build-fs.sh test
/root/.cargo/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
/bin/csh
Setting 'test' SGX, IAS and enclave values
  IAS_MODE: PROD
  SGX_MODE: HW
  CONSENSUS_ENCLAVE_CSS: /home/mobilecoin/full-service/target/release/consensus-enclave.css
  INGEST_ENCLAVE_CSS: /home/mobilecoin/full-service/target/release/ingest-enclave.css
building full service...
   Compiling typenum v1.17.0
   Compiling clang-sys v1.7.0
   Compiling protobuf v2.28.0
   Compiling mc-util-build-info v6.0.0 (/home/mobilecoin/full-service/mobilecoin/util/build/info)
   Compiling num-bigint v0.4.4
   Compiling crunchy v0.2.2
   Compiling hidapi v2.4.1
error: failed to run custom build command for `hidapi v2.4.1`

Caused by:
  process didn't exit successfully: `/home/mobilecoin/full-service/target/release/build/hidapi-0f1b6d5217db2a16/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=HIDAPI_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=HIDAPI_STATIC
  cargo:rerun-if-env-changed=HIDAPI_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=HIDAPI_STATIC
  cargo:rerun-if-env-changed=HIDAPI_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-freebsd
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_freebsd
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hidapi-2.4.1/build.rs:131:41:
  Unable to find hidapi: Could not run `PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 pkg-config --libs --cflags hidapi`
  The pkg-config command could not be found.

  Most likely, you need to install a pkg-config package for your OS.
  Try `apt install pkg-config`, or `yum install pkg-config`,
  or `pkg install pkg-config`, or `apk add pkgconfig` depending on your distribution.

  If you've already installed it, ensure the pkg-config command is one of the
  directories in the PATH environment variable.

  If you did not expect this build to link to a pre-installed system library,
  then check documentation of the hidapi crate for an option to
  build the library from source, or disable features or dependencies
  that require pkg-config.
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

Well, when posting this message, I noticed that the actual error is that I hadn't installed the pkg-config package (pkgconf on FreeBSD) and the hidapi package.

But I am now hitting this build error when building the package 820ish out of 924 (nearly there!!!):

 Compiling mc-util-parse v6.0.0 (/home/mobilecoin/full-service/mobilecoin/util/parse)
error[E0432]: unresolved import `crate::platform::PeripheralId`
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/api/mod.rs:44:5
   |
44 | use crate::platform::PeripheralId;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no `PeripheralId` in `platform`
   |
note: found an item that was configured out
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:6:77
   |
6  |     adapter::Adapter, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
   |                                                                             ^^^^^^^^^^^^
   = note: the item is gated behind the `linux` feature
note: found an item that was configured out
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:10:77
   |
10 |     adapter::Adapter, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
   |                                                                             ^^^^^^^^^^^^
note: found an item that was configured out
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:14:83
   |
14 |     adapter::Adapter, init, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
   |                                                                                   ^^^^^^^^^^^^
   = note: the item is gated behind the `android` feature
note: found an item that was configured out
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:18:77
   |
18 |     adapter::Adapter, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
   |                                                                             ^^^^^^^^^^^^
   = note: the item is gated behind the `windows` feature

error[E0432]: unresolved import `crate::platform::PeripheralId`
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/common/adapter_manager.rs:15:5
   |
15 | use crate::platform::PeripheralId;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no `PeripheralId` in `platform`
   |
note: found an item that was configured out
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:6:77
   |
6  |     adapter::Adapter, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
   |                                                                             ^^^^^^^^^^^^
   = note: the item is gated behind the `linux` feature
note: found an item that was configured out
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:10:77
   |
10 |     adapter::Adapter, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
   |                                                                             ^^^^^^^^^^^^
note: found an item that was configured out
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:14:83
   |
14 |     adapter::Adapter, init, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
   |                                                                                   ^^^^^^^^^^^^
   = note: the item is gated behind the `android` feature
note: found an item that was configured out
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:18:77
   |
18 |     adapter::Adapter, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
   |                                                                             ^^^^^^^^^^^^
   = note: the item is gated behind the `windows` feature

error[E0412]: cannot find type `Adapter` in this scope
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:29:18
   |
29 | assert_impl_all!(Adapter: Central, Clone, Debug, Send, Sized, Sync);
   |                  ^^^^^^^ not found in this scope

error[E0412]: cannot find type `Manager` in this scope
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:30:18
   |
30 | assert_impl_all!(Manager: api::Manager, Clone, Debug, Send, Sized, Sync);
   |                  ^^^^^^^ not found in this scope
   |
help: consider importing this trait
   |
21 + use crate::api::Manager;
   |

error[E0412]: cannot find type `Peripheral` in this scope
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:31:18
   |
31 | assert_impl_all!(Peripheral: api::Peripheral, Clone, Debug, Send, Sized, Sync);
   |                  ^^^^^^^^^^ not found in this scope
   |
help: consider importing this trait
   |
21 + use crate::api::Peripheral;
   |

error[E0412]: cannot find type `PeripheralId` in this scope
  --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/btleplug-0.10.5/src/platform.rs:33:5
   |
33 |     PeripheralId: Clone,
   |     ^^^^^^^^^^^^ not found in this scope

PS: Docker is not portable to FreeBSD so building from source is the only option we've got I'm afraid.

[nkosi23]

Here is the output of rustup show:

root@mobilecoinfullservicetest:/home/mobilecoin/full-service/tools # rustup show
Default host: x86_64-unknown-freebsd
rustup home:  /root/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-freebsd
nightly-2023-10-01-x86_64-unknown-freebsd
nightly-x86_64-unknown-freebsd (default)

active toolchain
----------------

nightly-2023-10-01-x86_64-unknown-freebsd (overridden by '/home/mobilecoin/full-service/rust-toolchain')
rustc 1.74.0-nightly (ca62d2c44 2023-09-30)

I am not a rust developer so I know very little about this ecosystem and what could cause the issue, but my guess is that the toolchain i use may be lacking capabilities.

[nkosi23]

Okay the issue is apparently in the platform.rs file of btleplug, there is no definition for the FreeBSD OS. I will try to submit a PR:

#[cfg(target_os = "linux")]
pub use crate::bluez::{
    adapter::Adapter, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
};
#[cfg(any(target_os = "macos", target_os = "ios"))]
pub use crate::corebluetooth::{
    adapter::Adapter, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
};
#[cfg(target_os = "android")]
pub use crate::droidplug::{
    adapter::Adapter, init, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
};
#[cfg(target_os = "windows")]
pub use crate::winrtble::{
    adapter::Adapter, manager::Manager, peripheral::Peripheral, peripheral::PeripheralId,
};

[nkosi23]

I have just submitted a PR containing a quick monkey patch to add a dummy FreeBSD definition reusing the crate for linux, which is guaranteed to cause a runtime error since FreeBSD's bluetooth stack is totally different.

But I'm crossing my fingers that no bluetooth capability is actually used, because this begs the question, why in the world does the full service API has a dependency on btleplug to start with? :confused:

There are very little chances that my ridiculous PR is going to be approved upstream now that I think about it :laughing: But applying this patch downstream is not going to be trivial as the rust toolchain downloads and builds crates automatically, I'd have to fiddle quite a bit to get going with the rust toolchain (edit: eventually came across patch-crate as I was digging into the source code and posting these messages). Even then, understanding how btleplug is being used by the full service API would be of the essence to ensure that this endeavour is even worth it (for if the package actually plays a critical role, then my workaround would not work).

The alternative and right approach would be to write proper upstream adapters for FreeBSD for btleplug, but this would require me to first learn the bluetooth spec, and then learn the bluetooth stack of FreeBSD. Oh and crash learn rust programming basics. One month of work at least :cry: Hopefully someone will reply that this package is being imported by mistake :smile:

[nkosi23]

Okay, progressing as I get my feet wet on the rust toolchain, the culprit is apparently ledger-lib, as per cargo tree see the last 10 lines:

── ledger-mob v0.16.0 (/home/mobilecoin/full-service/ledger-mob/lib)
│   ├── anyhow v1.0.79
│   ├── async-trait v0.1.77 (proc-macro) (*)
│   ├── base64 v0.21.7
│   ├── clap v4.4.18 (*)
│   ├── ed25519-dalek v2.1.0 (*)
│   ├── encdec v0.9.0
│   │   ├── encdec-base v0.9.0
│   │   │   ├── byteorder v1.5.0
│   │   │   ├── heapless v0.7.17
│   │   │   │   ├── hash32 v0.2.1
│   │   │   │   │   └── byteorder v1.5.0
│   │   │   │   ├── spin v0.9.8
│   │   │   │   │   └── lock_api v0.4.11 (*)
│   │   │   │   └── stable_deref_trait v1.2.0
│   │   │   │   [build-dependencies]
│   │   │   │   └── rustc_version v0.4.0 (*)
│   │   │   ├── num-traits v0.2.17 (*)
│   │   │   └── thiserror v1.0.56
│   │   │       └── thiserror-impl v1.0.56 (proc-macro)
│   │   │           ├── proc-macro2 v1.0.78 (*)
│   │   │           ├── quote v1.0.35 (*)
│   │   │           └── syn v2.0.48 (*)
│   │   └── encdec-macros v0.9.0 (proc-macro)
│   │       ├── darling v0.14.4
│   │       │   ├── darling_core v0.14.4
│   │       │   │   ├── fnv v1.0.7
│   │       │   │   ├── ident_case v1.0.1
│   │       │   │   ├── proc-macro2 v1.0.78 (*)
│   │       │   │   ├── quote v1.0.35 (*)
│   │       │   │   ├── strsim v0.10.0
│   │       │   │   └── syn v1.0.109
│   │       │   │       ├── proc-macro2 v1.0.78 (*)
│   │       │   │       ├── quote v1.0.35 (*)
│   │       │   │       └── unicode-ident v1.0.12
│   │       │   └── darling_macro v0.14.4 (proc-macro)
│   │       │       ├── darling_core v0.14.4 (*)
│   │       │       ├── quote v1.0.35 (*)
│   │       │       └── syn v1.0.109 (*)
│   │       ├── encdec-base v0.8.3
│   │       │   ├── byteorder v1.5.0
│   │       │   └── num-traits v0.2.17 (*)
│   │       ├── proc-macro2 v1.0.78 (*)
│   │       ├── quote v1.0.35 (*)
│   │       └── syn v1.0.109 (*)
│   ├── futures v0.3.30
│   │   ├── futures-channel v0.3.30 (*)
│   │   ├── futures-core v0.3.30
│   │   ├── futures-executor v0.3.30 (*)
│   │   ├── futures-io v0.3.30
│   │   ├── futures-sink v0.3.30
│   │   ├── futures-task v0.3.30
│   │   └── futures-util v0.3.30 (*)
│   ├── hex v0.4.3 (*)
│   ├── lazy_static v1.4.0
│   │   └── spin v0.5.2
│   ├── ledger-lib v0.1.0
│   │   ├── async-trait v0.1.77 (proc-macro) (*)
│   │   ├── btleplug v0.10.5
│   │   │   ├── async-trait v0.1.77 (proc-macro) (*)
│   │   │   ├── bitflags v1.3.2
│   │   │   ├── dashmap v5.5.3
│   │   │   │   ├── cfg-if v1.0.0
│   │   │   │   ├── hashbrown v0.14.3 (*)

And the btleplug dependency looks quite central to what this library is doing :disappointed: :

A rust-based library for interacting with Ledger hardware wallets. This provides low-level USB/HID, BLE, and TCP/Speculos Transports as well as a high level LedgerProvider interface that manages device connections using a pinned worker thread for use from async / tokio contexts.

Why is this library needed by the full service API? Any chance to put it behind a platform-specific compile flag (using the [cfg(target_os = "linux")] attribute)?

Maybe the build of ledger-mob itself could be put behind an OS flag since apparently it is specific to Ledger's hardware so I am not sure why including it in the full-service API is even relevant:

A MobileCoin NanoApp for Ledger nanosplus and nanox devices. You can grab the latest (unsigned) firmware and tooling here, or follow the Getting Started instructions to build your own.

[nick-mobilecoin]

Any chance to put it behind a platform-specific compile flag

I did a little digging around, it looks like the ledger logic is spread around a bit in the code base. So it would likely take a bit to isolate the ledger code.

My vote would be that it goes behind a feature flag maybe something like hw_signing or similar. Unfortunately I don't think this will get prioritized from our end at this time.

[nkosi23]

Oh sad to hear that. As a gentle feedback, working on adding mobilecoin support to our non-crypto-focused product has been constantly pushed back due to perceived and now real cans of worms.

Initially, based on the documentation, we assumed that it was not possible to build the full-service API without having an intel processor and the SGX libraries installed. Since we exclusively use FreeBSD for our workloads, and AMD processors, needless to say that we weren't very eager to get into this :smiling_face_with_tear: I was surprised when it turned out that the build actually worked fine without SGX for the reason you explained.

Now this ledger thing will be the actual show stopper. I guess the "easiest" course of action for us would be to provision a linux VM somewhere, which is not totally trivial as it also has implications on things like sysadmin (we'll now have 2 operating systems to run, update, secure, backup etc...). In our case, we can't use bhyve (FreeBSD's virtualizer allowing one to run VM's as easily as jails hosted by a FreeBSD host) which would have made things easier but our FreeBSD hosts are already running virtualized under a KVM managed by our hosting partner.

[nkosi23]

My vote would be that it goes behind a feature flag maybe something like hw_signing or similar. Unfortunately I don't think this will get prioritized from our end at this time.

I am not in a position to commit to working on it right now but to have visibility: would you welcome a PR isolating the ledger code, or would you prefer the core team to handle it itself (for example because there'd be too many implications on design, architecture etc...)

[holtzman]

as a quick experiment, would you see if the experiment/no-hardware-wallet branch builds on freebsd? It's building on MacOS sans the ledger-mob submodule, though I haven't really tested it.

As Nick said, it would be hard for us to prioritize refactoring things right now, so just gathering data...

[nkosi23]

@holtzman Many thanks for the experiment,

I am hitting a panic when building the grpcio-sys crate so I have opened an issue in their repo. However since this happens when building the crate 851 out 890, I have a sense that the rest of the build will work fine once the issue with grpc is addressed upstream.

The way you isolated ledger seems to work, but I'll need to run it to confirm this once the build issue is fixed.

[holtzman]

@nkosi23 I have removed the bug label as full-service is not supported for freebsd.

[holtzman]

@nkosi23, fyi, I created a freebsd 14 instance on EC2 and it built grpcio-sys. Perhaps there are packages in freebsd 13 that could be upgraded to get you past that?

Now I'm stuck at linking. There appear to be encryption-related symbols that sqlite3 is using but the linker is not finding. Full-service will optionally encrypt the wallet-db...

          >>> referenced by sqlite3.c:106013 (sqlcipher/sqlite3.c:106013)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_MAC_CTX_new
          >>> referenced by sqlite3.c:106020 (sqlcipher/sqlite3.c:106020)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_MAC_init
          >>> referenced by sqlite3.c:106029 (sqlcipher/sqlite3.c:106029)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib
          >>> referenced by sqlite3.c:106043 (sqlcipher/sqlite3.c:106043)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib
          >>> referenced by sqlite3.c:106036 (sqlcipher/sqlite3.c:106036)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_MAC_update
          >>> referenced by sqlite3.c:106054 (sqlcipher/sqlite3.c:106054)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib
          >>> referenced by sqlite3.c:106061 (sqlcipher/sqlite3.c:106061)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_MAC_final
          >>> referenced by sqlite3.c:106068 (sqlcipher/sqlite3.c:106068)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib
          >>> referenced by sqlite3.c:106074 (sqlcipher/sqlite3.c:106074)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_MAC_CTX_free
          >>> referenced by sqlite3.c:106087 (sqlcipher/sqlite3.c:106087)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_MAC_free
          >>> referenced by sqlite3.c:106088 (sqlcipher/sqlite3.c:106088)
          >>>               sqlite3.o:(sqlcipher_openssl_hmac) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_CIPHER_get_nid
          >>> referenced by sqlite3.c:106186 (sqlcipher/sqlite3.c:106186)
          >>>               sqlite3.o:(sqlcipher_openssl_get_cipher) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_CIPHER_get_key_length
          >>> referenced by sqlite3.c:106190 (sqlcipher/sqlite3.c:106190)
          >>>               sqlite3.o:(sqlcipher_openssl_get_key_sz) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_CIPHER_get_iv_length
          >>> referenced by sqlite3.c:106194 (sqlcipher/sqlite3.c:106194)
          >>>               sqlite3.o:(sqlcipher_openssl_get_iv_sz) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_CIPHER_get_block_size
          >>> referenced by sqlite3.c:106198 (sqlcipher/sqlite3.c:106198)
          >>>               sqlite3.o:(sqlcipher_openssl_get_block_sz) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib

          ld: error: undefined symbol: EVP_MD_get_size
          >>> referenced by sqlite3.c:0 (sqlcipher/sqlite3.c:0)
          >>>               sqlite3.o:(sqlcipher_openssl_get_hmac_sz) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib
          >>> referenced by sqlite3.c:0 (sqlcipher/sqlite3.c:0)
          >>>               sqlite3.o:(sqlcipher_openssl_get_hmac_sz) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib
          >>> referenced by sqlite3.c:0 (sqlcipher/sqlite3.c:0)
          >>>               sqlite3.o:(sqlcipher_openssl_get_hmac_sz) in archive /home/ec2-user/dev/full-service/target/release/deps/liblibsqlite3_sys-b42aee77f4cad872.rlib
          cc: error: linker command failed with exit code 1 (use -v to see invocation)

[nkosi23]

@holtzman Mmh have you installed the openssl package?

Apparently the issue I was hitting was due to the need to build gprcio against master because of an issue with the crate bindgen < 0.61 that has apparently been fixed on master.

I have been able to build the grpcio-sys dependency successfully using the latest nightly rust toolchain, but now I am hitting the following build error near the end of the build process :

warning: `mc-t3-connection` (lib) generated 1 warning
   Compiling mc-validator-connection v2.10.2 (/home/mobilecoin/full-service-experiment/full-service/validator/connection)
error[E0308]: mismatched types
  --> validator/connection/src/error.rs:30:59
   |
30 |             Error::Rpc(src) => mc_connection::Error::Grpc(src),
   |                                -------------------------- ^^^ expected `grpcio::error::Error`, found `grpcio::Error`
   |                                |
   |                                arguments to this enum variant are incorrect
   |
   = note: `grpcio::Error` and `grpcio::error::Error` have similar names, but are actually distinct types
note: `grpcio::Error` is defined in crate `grpcio`
  --> /home/mobilecoin/grpcio/grpc-rs/src/error.rs:17:1
   |
17 | pub enum Error {
   | ^^^^^^^^^^^^^^
note: `grpcio::error::Error` is defined in crate `grpcio`
  --> /root/.cargo/git/checkouts/grpc-rs-3d08006673990e5f/b82c902/src/error.rs:17:1
   |
17 | pub enum Error {
   | ^^^^^^^^^^^^^^
   = note: perhaps two different versions of crate `grpcio` are being used?
note: tuple variant defined here
  --> /home/mobilecoin/full-service-experiment/full-service/mobilecoin/connection/src/error.rs:28:5
   |
28 |     Grpc(GrpcError),
   |     ^^^^

error[E0599]: no method named `connect_to_uri` found for struct `ChannelBuilder` in the current scope
   --> validator/connection/src/lib.rs:70:14
    |
67  |           let ch = ChannelBuilder::new(env)
    |  __________________-
68  | |             .max_receive_message_len(std::i32::MAX)
69  | |             .max_send_message_len(std::i32::MAX)
70  | |             .connect_to_uri(uri, &logger);
    | |_____________-^^^^^^^^^^^^^^
    |
help: there is a method `connect` with a similar name, but with different arguments
   --> /home/mobilecoin/grpcio/grpc-rs/src/channel.rs:499:5
    |
499 |     pub fn connect(mut self, addr: &str) -> Channel {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
   --> validator/connection/src/lib.rs:222:53
    |
222 |             .get_last_block_info_opt(&Empty::new(), common_headers_call_option(&self.chain_id))
    |              -----------------------                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `grpcio::call::client::CallOption`, found `CallOption`
    |              |
    |              arguments to this method are incorrect
    |
    = note: `CallOption` and `grpcio::call::client::CallOption` have similar names, but are actually distinct types
note: `CallOption` is defined in crate `grpcio`
   --> /home/mobilecoin/grpcio/grpc-rs/src/call/client.rs:37:1
    |
37  | pub struct CallOption {
    | ^^^^^^^^^^^^^^^^^^^^^
note: `grpcio::call::client::CallOption` is defined in crate `grpcio`
   --> /root/.cargo/git/checkouts/grpc-rs-3d08006673990e5f/b82c902/src/call/client.rs:37:1
    |
37  | pub struct CallOption {
    | ^^^^^^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `grpcio` are being used?
note: method defined here
   --> /home/mobilecoin/full-service-experiment/full-service/target/release/build/mc-consensus-api-99e1db21a1f0ba5b/out/protos-auto-gen/consensus_common_grpc.rs:45:12
    |
45  |     pub fn get_last_block_info_opt(&self, req: &super::empty::Empty, opt: ::grpcio::CallOption) -> ::grpcio::Result<super::consensus_common::LastBlockInfoResponse> {
    |            ^^^^^^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
   --> validator/connection/src/lib.rs:238:53
    |
238 |             .get_last_block_info_opt(&Empty::new(), common_headers_call_option(&self.chain_id))
    |              -----------------------                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `grpcio::call::client::CallOption`, found `CallOption`
    |              |
    |              arguments to this method are incorrect
    |
    = note: `CallOption` and `grpcio::call::client::CallOption` have similar names, but are actually distinct types
note: `CallOption` is defined in crate `grpcio`
   --> /home/mobilecoin/grpcio/grpc-rs/src/call/client.rs:37:1
    |
37  | pub struct CallOption {
    | ^^^^^^^^^^^^^^^^^^^^^
note: `grpcio::call::client::CallOption` is defined in crate `grpcio`
   --> /root/.cargo/git/checkouts/grpc-rs-3d08006673990e5f/b82c902/src/call/client.rs:37:1
    |
37  | pub struct CallOption {
    | ^^^^^^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `grpcio` are being used?
note: method defined here
   --> /home/mobilecoin/full-service-experiment/full-service/target/release/build/mc-consensus-api-99e1db21a1f0ba5b/out/protos-auto-gen/consensus_common_grpc.rs:45:12
    |
45  |     pub fn get_last_block_info_opt(&self, req: &super::empty::Empty, opt: ::grpcio::CallOption) -> ::grpcio::Result<super::consensus_common::LastBlockInfoResponse> {
    |            ^^^^^^^^^^^^^^^^^^^^^^^

error[E0277]: the trait bound `grpcio::Error: AttestationError` is not satisfied
   --> validator/connection/src/lib.rs:259:15
    |
259 |             })?;
    |               ^ the trait `AttestationError` is not implemented for `grpcio::Error`, which is required by `Result<u64, mc_connection::Error>: FromResidual<Result<std::convert::Infallible, grpcio::Error>>`
    |
    = help: the trait `AttestationError` is implemented for `ThickClientAttestationError`
    = note: required for `mc_connection::Error` to implement `From<grpcio::Error>`
    = note: required for `Result<u64, mc_connection::Error>` to implement `FromResidual<Result<std::convert::Infallible, grpcio::Error>>`

warning: unused import: `ConnectionUriGrpcioChannel`
  --> validator/connection/src/lib.rs:16:20
   |
16 | use mc_util_grpc::{ConnectionUriGrpcioChannel, CHAIN_ID_GRPC_HEADER};
   |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

Some errors have detailed explanations: E0277, E0308, E0599.
For more information about an error, try `rustc --explain E0277`.
warning: `mc-validator-connection` (lib) generated 1 warning
error: could not compile `mc-validator-connection` (lib) due to 5 previous errors; 1 warning emitted
warning: build failed, waiting for other jobs to finish...
warning: `mc-ledger-db` (lib) generated 1 warning
warning: `mc-watcher` (lib) generated 2 warnings
warning: `mc-fog-api` (lib) generated 1 warning
warning: `mc-mobilecoind` (lib) generated 4 warnings
    Building [=======================> ] 892/904: mc-mobilecoind-api   

[nick-mobilecoin]

My hunch is this

note: perhaps two different versions of crate grpcio are being used? note: method defined here

Those kind of errors, "expect name, got (same) name", are usually two different versions of the crate. If you look in your local Cargo.lock file you'll probably see 2 versions of grpcio** called out.

I'd be curious how you did the following

I have been able to build the grpcio-sys dependency successfully using the latest nightly rust toolchain

We may be calling out grpcio inconsistently and only one place was updated

[nkosi23]

Thanks a lot for your help! Indeed the Cargo.lock file contained multiple versions of grpcio and grpcio-sys but the issue is 100% from me.

Initially I wasn't aware that [patch.crates-io] existed (this is my first time toying around with rust and the tooling is blowing my mind, what a modern development toolset), so I was modifying the Cargo.toml files of individuals crates manually, forgot some, then experimented by pointing grpcio to a local folder then to git (forgetting to update .toml files along the way), so I'm not surprised this file became messy.

So, in light of your pointer, starting from the experiment branch, here are all the changes i eventually made:

• Add grpcio = { git = "https://github.com/tikv/grpc-rs.git", branch = "master" } to the [patch.crates-io] section of the top-level Cargo.toml
• replace the content of rust-toolchain with: nightly-2024-05-26
• Run cargo generate-lockfile

And I am glad to report that the build completed succesfully :tada: :tada: :fireworks: I am now trying to run it with run-fs.sh but I get a no such a file or directory error when doing so:

Setting 'test' environment values
  MC_CHAIN_ID: test
  MC_PEER: mc://node1.test.mobilecoin.com/,mc://node2.test.mobilecoin.com/
  MC_TX_SOURCE_URL: https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/,https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node2.test.mobilecoin.com/
  MC_FOG_INGEST_ENCLAVE_CSS: /root/.mobilecoin/test/ingest-enclave.css
  MC_WALLET_DB: /root/.mobilecoin/test/wallet-db/wallet.db
  MC_LEDGER_DB: /root/.mobilecoin/test/ledger-db
  MC_LISTEN_HOST: 127.0.0.1
Starting Full-Service Wallet from /root/.mobilecoin/test
./tools/run-fs.sh: line 204: /root/.mobilecoin/test/full-service: No such file or directory

The content of the build was put inside target/release folder of the git workspace so I am trying to figure out the symlinks that are needed (I built using build-fs.sh)

[nkosi23]

Okay, after adding the symlink:

ln -s /home/mobilecoin/full-service-experiment/full-service/target/release/full-service /root/.mobilecoin/test/

Everything works like a charm, the ledger is currently syncing :)

I will test that the main features needed by our app work and will keep you posted. Thanks a lot for your help with this

[nkosi23]

Oh I have a question, since we'll likely be stuck with the version available in the experiment branch for a while, are releases usually backward compatible? I mean at a protocol level, how likely is it that at some point the transactions created by our node become rejected?

The spirit of my question is: is it guaranteed to happen with every update - for example due to SGX attestation - or are such breaking changes relatively rare.

[holtzman]

Good question!

TL; DR: full-service has few mandatory updates and they are far between. One is imminent, but we forked the experimental branch off main after the mandatory updates had landed, so that branch is already good to go.

When full-service attests the enclaves it uses, it does not require that their individual build measurements are on a whitelist compiled into full-service. This is exactly so that the same version of full-service can continue to work across enclave-side updates, as long as there are no breaking protocol changes. It does validate that the enclave was signed by the trusted authority, has the expected product id, and meets a minimum version requirement before trusting it.

In contrast, Sentz and Signal do validate that each enclave's build measurements are on a compiled-in whitelist. The environments of use, ease of updating, and education / contextual awareness level of the operator of each of these are very different and thus different policies are appropriate.

On occasion (3 times so far) we have had to make breaking changes in the protocol that have required full-service to be updated. The first was when we went multi-asset (~10/22); the second was phase 2 of multi-asset which introduced signed-contingent-inputs, allowing for atomic swaps between assets (~3/2023); and, the third will be to switch from SGX EPID attestation to SGX DCAP due to Intel phasing out EPID support (7/2024).

The timing of this branch is very good for that 3rd one as we are mid-way through the deployment process, and full-service is already updated and compatible with both the current and soon-to-be-deployed enclaves. In fact, testnet is already upgraded to v6.0, which is scheduled to be deployed to mainnet the week of July 8.

[holtzman]

Inspired by the topic, I merged main back into this branch as we've done a bunch of feature updates since the branch forked.

I built and tested on MacOS, including a scripted client test that end-to-end exercises a running full-service via its api.

[nkosi23]

Hi gentlemen! I am getting plenty of "transaction fetch failure" on the testnet but the mainnet is not affected for time being:

2024-10-16 06:20:46.583422216 UTC INFO Worker 0 failed getting transactions for block 3215362: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311002.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:20:47.804507425 UTC INFO Worker 0 failed getting transactions for block 3215362: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311002.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:20:50.125419341 UTC INFO Appending 2 blocks to ledger, which currently has 3215362 blocks, mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:225
2024-10-16 06:21:46.823191983 UTC INFO Worker 1 failed getting transactions for block 3215364: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311004.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:21:48.032189381 UTC INFO Worker 1 failed getting transactions for block 3215364: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311004.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:21:50.421705885 UTC INFO Appending 2 blocks to ledger, which currently has 3215364 blocks, mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:225
2024-10-16 06:22:46.755255386 UTC INFO Worker 0 failed getting transactions for block 3215366: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311006.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:22:47.947339101 UTC INFO Worker 0 failed getting transactions for block 3215366: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311006.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:22:50.287187850 UTC INFO Appending 2 blocks to ledger, which currently has 3215366 blocks, mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:225
2024-10-16 06:23:46.823240720 UTC INFO Worker 0 failed getting transactions for block 3215368: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311008.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:23:48.030083856 UTC INFO Worker 0 failed getting transactions for block 3215368: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311008.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:23:50.323399914 UTC INFO Appending 2 blocks to ledger, which currently has 3215368 blocks, mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:225
2024-10-16 06:24:46.834016309 UTC INFO Worker 1 failed getting transactions for block 3215371: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/000000000031100b.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:24:48.058182625 UTC INFO Worker 1 failed getting transactions for block 3215371: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/000000000031100b.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:24:50.304910394 UTC INFO Appending 2 blocks to ledger, which currently has 3215370 blocks, mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:225
2024-10-16 06:25:47.011266882 UTC INFO Worker 0 failed getting transactions for block 3215372: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/000000000031100c.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:25:48.240849646 UTC INFO Worker 0 failed getting transactions for block 3215372: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/000000000031100c.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:25:50.566475900 UTC INFO Appending 2 blocks to ledger, which currently has 3215372 blocks, mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:225
2024-10-16 06:26:46.767837035 UTC INFO Worker 1 failed getting transactions for block 3215374: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/000000000031100e.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:26:47.953534282 UTC INFO Worker 1 failed getting transactions for block 3215374: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/000000000031100e.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:26:50.321579868 UTC INFO Appending 2 blocks to ledger, which currently has 3215374 blocks, mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:225
2024-10-16 06:27:46.780521036 UTC INFO Worker 1 failed getting transactions for block 3215376: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311010.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:27:47.965405197 UTC INFO Worker 1 failed getting transactions for block 3215376: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311010.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:27:50.278216309 UTC INFO Appending 2 blocks to ledger, which currently has 3215376 blocks, mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:225
2024-10-16 06:28:46.893224725 UTC INFO Worker 0 failed getting transactions for block 3215378: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311012.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:28:48.120526742 UTC INFO Worker 0 failed getting transactions for block 3215378: Transaction fetch failure: InvalidBlockReceived("https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.test.mobilecoin.com/00/00/00/00/00/31/10/0000000000311012.pb", "protobuf parse failed: WireError(UnexpectedWireType(WireTypeEndGroup))"), mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:711
2024-10-16 06:28:50.422202513 UTC INFO Appending 2 blocks to ledger, which currently has 3215378 blocks, mc.app: full-service, mc.module: mc_ledger_sync::ledger_sync::ledger_sync_service, mc.src: mobilecoin/ledger/sync/src/ledger_sync/ledger_sync_service.rs:225

I am running the full-service API version 2.10.2

[nick-mobilecoin]

Thanks for the heads up, looks like testnet node1 has been having some issues since about Oct 8th. one can see it's signature is missing from the block explorer starting at, 3193708 We're currently looking into it