Fresh install - build fail mmap issue

Hraesvel commented 1 year ago

Checklist

[X] I've looked through the issues and pull requests for similar reports

Describe your issue

Hello, this my time using cross to compile to Linux from Mac (intel). Using Podman as my vm engine. However the project fails to build even for a simple hello world executable.

What target(s) are you cross-compiling for?

No response

Which operating system is the host (e.g computer cross is on) running?

[X] macOS
[ ] Windows
[ ] Linux / BSD
[ ] other OS (specify in description)

What architecture is the host?

[X] x86_64 / AMD64
[ ] arm32
[ ] arm64 (including Mac M1)

What container engine is cross using?

[ ] docker
[X] podman
[ ] other container engine (specify in description)

cross version

cross 0.2.5 (37c681a 2023-07-17)

Example


info: self-update is disabled for this build of rustup
info: any updates to rustup will need to be fetched with your system package manager
info: downloading component 'rust-src'
info: installing component 'rust-src'
   Compiling rust-woof v0.1.0 (/Users/username/rust-woof)
error[E0786]: found invalid metadata files for crate `std`
  |
  = note: failed to mmap file '/Users/username/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-7b9f6349d87c69a1.rlib': No such device (os error 19)
  = note: failed to mmap file '/Users/username/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-7b9f6349d87c69a1.so': No such device (os error 19)
  = note: failed to mmap file '/Users/username/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-e299e57d9e5c1111.rlib': No such device (os error 19)

error: cannot find macro `println` in this scope
 --> src/main.rs:2:5
  |
2 |     println!("Hello, world!");
  |     ^^^^^^^

error: requires `sized` lang_item

For more information about this error, try `rustc --explain E0786`.
error: could not compile `rust-woof` (bin "rust-woof") due to 3 previous errors

Additional information / notes

No response

Emilgardis commented 1 year ago

can you do the following, make sure to use the proper path to the .rustup folder

podman run -v /Users/username/.rustup/toolchains/:/rustup --rm -it ubuntu:20.04 ls /rustup/stable-x86_64-unknown-linux-gnu/lib/rustlib/

to me, it seems like the podman machine can't read the folder.

gl-yziquel commented 8 months ago

I have the same issue with a different setup. I've been using incus/lxd to mount a virtual machine where I have rust code. I changed the --cache option of virtiofsd from "auto" to "never" (which made it today in the zabbly daily build of incus), and I get the exact same errors. On code that used to compile fine.

There are other weirdnesses to my setup, so I cannot claim that the above is all that matters. But I do get the same errors (with a ZFS filesystem on the host being mounted with virtiofsd, and bind --mounts on the guest).

However, something tells me that while this may be an OS+virtualisation issue, rust itself seems to be at fault too. Because I do get the "No such device (os error 19)" in another context on that same virtual machine: When cloning a repo, I tried "git clone" with the usual git tool and "gix clone" with the gix cli of the gitoxyde project (which is a somewhat-drop-in replacement to git, written in rust). "git clone" clones fine. "gix clone" throws "No such device (os error 19)".

It may be that rust is trying to be overly smart in some ways, and shoots itself in the foot. If that is correct, then it may not be a VM or container problem. It may be, or may not.

minime@virtucon ~> gix clone https://github.com/cross-rs/cross
[...]
Error: An IO error occurred when reading the pack or creating a temporary file

Caused by:
    No such device (os error 19)

The same git clone works fine, however.

Update:

I tried the gix clone command in two places in my virtual machine.

In a place that is backed by the rootfs zfs storage of incus/lxc: gix clone works fine.
In a place that is backed by a zfs share over virtiofsd + mount --bind on the guest: gix clone throws OS error 19.
In a place that is backed by a zfs share over virtiofsd without a mount --bind on the guest: gix clone throws OS error 19.

Conclusion: mount --bind is not the likely culprit. Only the virtiofsd filesystem. With --cache=auto on virtiofsd, I did not have this problem. With --cache=never on virtiofsd, I start having this problem.

Strace:

strace gix clone blah blah blah give me the following:

openat(AT_FDCWD, "/path/to/my/.git/objects/pack/.tmpoTgZcj, O_RDONLY|O_CLOEXEC) = 8
statx(8, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0600, stx_size=3298835, ...}) = 0
mmap(NULL, 3298835, PROT_READ, MAP_SHARED, 8, 0) = -1 ENODEV (No such device)
close(8) = 0

A similar strace but for git instead of gix yields something slightly different:

openat(AT_FDCWD, "/path/to/my/.git/objects/pack/pack-fb429f8a8e65eced68e89f4f1bae9bb1460f23f4.idx", O_RDO NLY|O_CLOEXEC) = 4
newfstatat(4, "", {st_mode=S_IFREG|0444, st_size=302800, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 302800, PROT_READ, MAP_PRIVATE, 4, 0) = 0x708a59d6b000
close(4) = 0

This first likely culprit is the use of MAP_SHARED instead of MAP_PRIVATE.

Emilgardis commented 7 months ago

I think what you've identified should be raised as an issue on rust-lang/rust, the problem seems to be out of our hands and the culprit is the backend for OpenOptions/fs in rust?

gl-yziquel commented 7 months ago

On #irc, the rust people told me it's not a rust problem, but rather a virtiofs problem. They may be right. They may be wrong. It may involve the DAX window for memory mapped files with virtiofs. And it may be a similar issue with docker. I can't tell. But it also is tied to rust's safety expectations, and many implementations of memory mapped files, they told me, tend to use MAP_SHARED instead of MAP_PRIVATE by default. Which would explain the subtle difference observed between git and gix.

All in all, I do not really know where to start with that kind of issue.

I tend to believe it's a problem pertaining to immature DAX window support in virtualisation technologies.

But don't quote me on that.

gl-yziquel commented 7 months ago

It is conceivable that the following issue in the rustc compiler will solve this one.

https://github.com/rust-lang/rust/issues/122262

With a bit of luck.

gl-yziquel commented 7 months ago

I think what you've identified should be raised as an issue on rust-lang/rust, the problem seems to be out of our hands and the culprit is the backend for OpenOptions/fs in rust?

There are some people on the issue and PR I opened who claim that a bug in a filesystem is not a bug in rust. Which is kind of true, but on the other hand, it is the job of a compiler not to bail out in a virtualised setting.

But just so you know: some people believe the problem is with your filesystem.

I think that position is disingenuous.

I hope this doesn't hold back my PR which, I believe, will solve this issue.

cross-rs / cross