Can't use `insteadOF` for fetching private crates via SSH instead of HTTPs

redeexpressos commented 4 months ago

Problem

So, I'm in a custom shell (bitbake) generated by kas where I'm building recipes with meta-rust-bin that download Rust code and build them for a custom platform. I have a Rust bin A that depends on a private crate B. A has a .cargo/config with git-fetch-with-cli = true In this shell, I cannot login manually, so everything has to be pre-set for it to fetch. Problem I am running into:

A cannot find the private crate B via https, probably because the shell is not logged in to git, so no authorization.
Ive tried the insteadOf method, so: git config --global url."ssh://git@workspace.com/B.git".insteadOf "https://workspace.com/B.git"
- With this, I can clone with a https link and it does via ssh, which is GOOD.

However, with Cargo, it's asking for some key. Im not sure if its the right place to ask, but might be some Cargo trick im failing. Logs:

Log data follows:
| DEBUG: Executing shell function do_compile
| NOTE: which rustc: /home/ubuntu/user/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/A/1.0+gitAUTOINC+8c7391c38b-r0/recipe-sysroot-native/usr/bin/rustc
| NOTE: rustc --version rustc 1.79.0 (129f3b996 2024-06-10)
| NOTE: which cargo: /home/ubuntu/user/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/A/1.0+gitAUTOINC+8c7391c38b-r0/recipe-sysroot-native/usr/bin/cargo
| NOTE: cargo --version cargo 1.79.0 (ffa9cf99a 2024-06-03)
| NOTE: cargo build --verbose --manifest-path /home/ubuntu/user/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/A/1.0+gitAUTOINC+8c7391c38b-r0/git/Cargo.toml --target=armv7-unknown-linux-gnueabihf --profile=release
|     Updating crates.io index
|     Updating git repository `https://workspace.com/git/B.git`
| error: failed to get `B` as a dependency of package `A v0.1.0 (/home/ubuntu/user/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/A/1.0+gitAUTOINC+8c7391c38b-r0/git)`
| 
| Caused by:
|   failed to load source for dependency `B`
| 
| Caused by:
|   Unable to update https://workspace.com/git/B.git?rev=ef257ddd990fa016533542be516f4ce7cd8bb32d#ef257ddd
| 
| Caused by:
|   failed to fetch into: /home/ubuntu/user/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/A/1.0+gitAUTOINC+8c7391c38b-r0/cargo_home/git/db/B-cfc1139fb69d89ce
| 
| Caused by:
|   error: unknown SSH host key
|   The SSH host key for `workspace.com` is not known and cannot be validated.
| 
|   To resolve this issue, add the host key to the `net.ssh.known-hosts` array in your Cargo configuration (such as /home/ubuntu/user/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/A/1.0+gitAUTOINC+8c7391c38b-r0/cargo_home/config.toml) or in your OpenSSH known_hosts file at /tmp/tmpk8pzrsyh/.ssh/known_hosts
| 
|   The key to add is:
| 
|   workspace.com ecdsa-sha2-nistp256 <removed>
| 
|   The ECDSA key fingerprint is: SHA256:<removed>
|   This fingerprint should be validated with the server administrator that it is correct.
| 
|   See https://doc.rust-lang.org/stable/cargo/appendix/git-authentication.html#ssh-known-hosts for more information.
| WARNING: exit code 101 from a shell command.

The obvious answer would be to, read logs... Yes, but Im trying to understand if there's a way of automating this.

Steps

Hard to reproduce, since its a custom Yocto Bitbake shell generated by kas But TLDR: Im able to git clone a private repository after insteadOF from http to ssh, but Cargo is not able to fetch it as a crate dependency.

Possible Solution(s)

No response

Notes

No response

Version

No response

weihanglo commented 4 months ago

If you've set net.git-fetch-with-cli = true correctly, you shouldn't have hit the code path.

I saw the log containing the cargo invocation with --manifest-path. Chances are that the Cargo configuration file .cargo/config.toml was not loaded correctly. They need to follow the hierarchical structure for being loaded.

What you can investigate for now is:

Check the working directory of the shell invoking Cargo has the correct .cargo/config.toml in cwd or any parent directories.
Set env var CARGO_NET_GIT_FETCH_WITH_CLI and see if it works.

redeexpressos commented 4 months ago

So: --manifest-path /home/ubuntu/user/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/A/1.0+gitAUTOINC+8c7391c38b-r0/git/Cargo.toml

ubuntu@ubuntu:/home/ubuntu/user/builder/build $ ls -al /home/ubuntu/userbuilder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/A/1.0+gitAUTOINC+8c7391c38b-r0/git/
.cargo .git .gitignore Cargo.lock  Cargo.toml  Cross.toml  debug readme.md  src

So .cargo is there. And it has the correct config with

[net]
git-fetch-with-cli = true

Tried setting CARGO_NET_GIT_FETCH_WITH_CLI to true, still no success.

redeexpressos commented 4 months ago

If I run the manifest command outside the shell, I can cargo build as normal.

redeexpressos commented 4 months ago

If helps, I'm using this to build: https://github.com/rust-embedded/meta-rust-bin/blob/master/classes/cargo_bin.bbclass

redeexpressos commented 4 months ago

Looks like adding netrc authentication and:

do_compile:prepend() {
    export CARGO_NET_GIT_FETCH_WITH_CLI=true
}

fixed the issue.

For some reason .cargo/config is not doing what its supposed in this shell..

weihanglo commented 4 months ago

It seems that the current directory is different from the Cargo package root path

/home/ubuntu/user/builder/build
/home/ubuntu/userbuilder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/A/1.0+gitAUTOINC+8c7391c38b-r0/git/

Cargo only search .cargo/config.toml from cwd to all parent directories. If that is your cwd than it is expected, as early mentioned in a comment.

rust-lang / cargo