jackady dev command fails

[gotjoshua]

Describe the bug

./jackady -v dev -d

... all the build stuff is green up to here ...
🟢 Preparing daemon runtime ⏱️  4ms
🟢 Running the ipfsd binary /mnt/stripe/gotjosh/dev/nutria/prebuilts/x86_64-unknown-linux-gnu/ipfsd in /mnt/stripe/gotjosh/dev/nutria/builder/output/dev with --cfg /mnt/stripe/gotjosh/dev/nutria/defaults/ipfsd.toml
🔴 Command failed: Desktop command error: Io error: No such file or directory (os error 2)

if i run /mnt/stripe/gotjosh/dev/nutria/prebuilts/x86_64-unknown-linux-gnu/ipfsd i get:

fish: Job 1, '/mnt/stripe/gotjosh/dev/nutria/…' terminated by signal SIGSEGV (Address boundary error)

Environment

• Desktop Linux (NixOS)

To Reproduce Steps to reproduce the behavior: Follow instructions from readme

Expected behavior the app to run in dev mode

Additional context I assume some traditional linux libs/binaries are missing ( as is often the case with NixOS), but not sure how to find out which ones

[gotjoshua]

just noticed that the ipfsd is a prebuilt that seems to be built for debian...

i assume it expects things to be in a certain way, that just aren't in NixOS

[fabricedesre]

Hi, thanks for the report!

You're correct that ipfs is build on a debian based (ubuntu LTS) machine. Do you know what should be done to support NixOS?

[gotjoshua]

am i correct to assume that it is a build of your beetle fork? https://github.com/capyloon/beetle

i see that there is a dockerfile for a distroless iroh-one https://github.com/capyloon/beetle/blob/main/docker/Dockerfile.iroh-one

could be useful in these efforts

[fabricedesre]

Yes, this is a build of iroh-one from this beetle fork. If you can confirm that the docker build fixes the crash on Nix I'll switch to it.

[gotjoshua]

i wonder if adding it as an additional target could make sense...

i'll let you know if i manage to confirm that

[fabricedesre]

Is NixOS musl-based like Alpine?

[gotjoshua]

I think I can't really do justice to describing what nixos is. Minimalistic, deterministic, declarative...

This repo has quite a few links at the top https://github.com/NixOS/mobile-nixos

I send that one as I think it could be a very interesting experiment to get capyloon running on a nixos based mobile device !

But first I'd like to see it in dev mode on my desktop... Thanks for your engagement!

[fabricedesre]

Btw I looked a the beetle docker files and they pull a rust image which is debian based. Not sure if this would produce a binary that works on NixOS

[gotjoshua]

From what I understood, it uses the debian distro to build and then creates a distroless and copies in the build results Have a look a bit lower down: FROM gcr.io/distroless/cc

Unfortunately I didn't get around to test before traveling, maybe over the weekend

[tennox]

Hi, a friend of @gotjoshua here. I'm not sure what causes the SIGSEGV, do you have an idea?

Missing gcc was my first thought, but that would look like this (in docker alpine):

/app # ./prebuilts/x86_64-unknown-linux-gnu/ipfsd
/bin/sh: ./prebuilts/x86_64-unknown-linux-gnu/ipfsd: not found

(note: in docker debian the same binary works fine)

Short NixOS explanation

Roughly speaking, in NixOS, instead of the OS providing dependencies (like gcc/protoc), each package specifies its' dependencies. So statically linked is one way to not have external dependencies, or we'd need to install the the package the nix way, specifying a way to build it and its' dependencies - can explain more if interested & I already started a nix flake definition, but:

iroh-one

I managed to set up both: the linked docker image (although it was weirdly using COPY ../ which is not allowed in the docker I know, so I had to change that & some other things :confused:) and a nix flake for building iroh-one, but I'm failing with both strategies at compiling the dependencies:

   Compiling trust-dns-proto v0.22.0
   Compiling libp2p-quic v0.7.0-alpha
error[E0308]: mismatched types
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-quic-0.7.0-alpha/src/connection/connecting.rs:85:44
   |
85 |                     return Poll::Ready(Ok((peer_id, self.connection.take().unwrap())));
   |                                            ^^^^^^^ expected `PeerId`, found `libp2p_identity::peer_id::PeerId`
   |
   = note: `libp2p_identity::peer_id::PeerId` and `PeerId` have similar names, but are actually distinct types
note: `libp2p_identity::peer_id::PeerId` is defined in crate `libp2p_identity`
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-identity-0.1.3/src/peer_id.rs:50:1
   |
50 | pub struct PeerId {
   | ^^^^^^^^^^^^^^^^^
note: `PeerId` is defined in crate `libp2p_core`
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-core-0.38.0/src/peer_id.rs:40:1
   |
40 | pub struct PeerId {
   | ^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-quic-0.7.0-alpha/src/endpoint.rs:81:73
   |
81 |         let client_tls_config = Arc::new(libp2p_tls::make_client_config(keypair, None).unwrap());
   |                                          ------------------------------ ^^^^^^^ expected `libp2p_identity::keypair::Keypair`, found `libp2p_core::identity::Keypair`
   |                                          |
   |                                          arguments to this function are incorrect
   |
   = note: `libp2p_core::identity::Keypair` and `libp2p_identity::keypair::Keypair` have similar names, but are actually distinct types
note: `libp2p_core::identity::Keypair` is defined in crate `libp2p_core`
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-core-0.38.0/src/identity.rs:68:1
   |
68 | pub enum Keypair {
   | ^^^^^^^^^^^^^^^^
note: `libp2p_identity::keypair::Keypair` is defined in crate `libp2p_identity`
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-identity-0.1.3/src/keypair.rs:58:1
   |
58 | pub enum Keypair {
   | ^^^^^^^^^^^^^^^^
note: function defined here
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-tls-0.1.0/src/lib.rs:43:8
   |
43 | pub fn make_client_config(
   |        ^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-quic-0.7.0-alpha/src/endpoint.rs:82:73
   |
82 |         let server_tls_config = Arc::new(libp2p_tls::make_server_config(keypair).unwrap());
   |                                          ------------------------------ ^^^^^^^ expected `libp2p_identity::keypair::Keypair`, found `libp2p_core::identity::Keypair`
   |                                          |
   |                                          arguments to this function are incorrect
   |
   = note: `libp2p_core::identity::Keypair` and `libp2p_identity::keypair::Keypair` have similar names, but are actually distinct types
note: `libp2p_core::identity::Keypair` is defined in crate `libp2p_core`
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-core-0.38.0/src/identity.rs:68:1
   |
68 | pub enum Keypair {
   | ^^^^^^^^^^^^^^^^
note: `libp2p_identity::keypair::Keypair` is defined in crate `libp2p_identity`
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-identity-0.1.3/src/keypair.rs:58:1
   |
58 | pub enum Keypair {
   | ^^^^^^^^^^^^^^^^
note: function defined here
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-tls-0.1.0/src/lib.rs:65:8
   |
65 | pub fn make_server_config(
   |        ^^^^^^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0308`.
error: could not compile `libp2p-quic` (lib) due to 3 previous errors
warning: build failed, waiting for other jobs to finish...
The command '/bin/sh -c cargo build --bin iroh-one --profile=docker' returned a non-zero code: 101

I noticed there is no Cargo.lock committed, so maybe you might be using a different version? Could you push that maybe? Or do you have different idea? :thinking:

I can push a reproduction repo if that helps :tipping_hand_man:

[fabricedesre]

@tennox You need to be on the capyloon branch of the beetle repo, where Cargo.lock is commited along with all the crates vendored under ./third-party to make builds reproducible.

[fabricedesre]

Also, please build with --features=http-uds-gateway such as:

pushd iroh-one
cargo build --release  --features=http-uds-gateway
popd

[tennox]

That helped :sweat_smile: :+1:

managed to nix build the binary after figuring out & defining quite some dependencies :sweat: (not an expert of dynamic linking and libs and so on) .. will post details tomorrow