`failed to unshare` i/o issues

[jhheider]

potentially interest data point:

with a test script like this:

$ git clone https://github.com/pkgxdev/pantry
Cloning into 'pantry'...
$ cpz pantry pantry2
Error: An I/O error occurred
│
╰─▶ Operation not permitted (os error 1)
    ╰╴Failed to unshare FD table.

i get errors (ref) on both linux/amd64 and linux/arm64, but not on macos; a simpler test passes both:

$ mkdir -p a/b/c/d/e
$ echo aaa > a/b/c/d/e/f
$ echo aaa > a/b/c/d/e/g
$ cpz a a2

further, i get an unshare error on linux/amd64 for rmz, regardless of complexity:

$ git clone https://github.com/pkgxdev/pantry
Cloning into 'pantry'...
$ test -d pantry
$ rmz pantry
Error: An I/O error occurred
│
╰─▶ Operation not permitted (os error 1)
    ╰╴Failed to unshare I/O.

$ mkdir -p a/b/c/d/e
$ touch a/b/c/d/e/f
$ test -d a
$ rmz a
Error: An I/O error occurred
│
╰─▶ Operation not permitted (os error 1)
    ╰╴Failed to unshare I/O.

(ref2)

i suspect that the rmz error might be specific to GitHub Actions runners, since the rest are self-hosted, and the cpz error is likely related to the complexity of a full git clone. but i thought it was interesting that the problems manifested solely on linux, so i thought i'd bring it to your attention. i'll update if i find anything else (might try using sudo to see if that makes a difference. or -f, but that seems unlikely.).

[jhheider]

it will remove a simple file, but seems to fail (at least on the GHA runner) with a nested structure. interestingly, it works fine in docker on my machine in linux/amd64. so, this might be a weird edge case, but i thought i'd bring them to your attention.

[jhheider]

the rmz issue appears to be intermittent on linux/arm64, actually: (ref). and that is a self-hosted docker image.

[SUPERCILEX]

Nice reverse engineering lol. So it only happens on linux and when there are directories involved because unshare is a linux syscall that I run whenever I start a new thread (which is needed for fast directory processing).

The problem is that docker blocks unshare, so you'll need to use seccomp to allow it: https://securitylabs.datadoghq.com/articles/container-security-fundamentals-part-6/ Technically we don't need unshare but it is a performance improvement. Maybe it makes sense to ignore errors? But I strongly dislike that because then if it doesn't work when it should, we'll have no idea.

[jhheider]

ah, there you go. hm. possibly a flag to ignore (or force?) and a warning about docker as a potential issue on failure? then, at least, docker users could do something like alias rmz='/path/to/rmz --ignore-unshare-failure'?

[SUPERCILEX]

Maybe? I'm trying to decide if this is my problem or a docker config problem. It's weird that docker blocks unshare by default, especially considering big names like GitHub Actions don't, so I really do feel like this is just a bad docker config. But I also think a hidden environment variable could be reasonable. Maybe NO_UNSHARE and it just won't even try? Still would prefer docker just allowing unshared though.

[vuppalaprasanth]

if multiple rmz on same folder, everything fails with

Error: An I/O error occurred
│
╰─▶ No such file or directory (os error 2)
    ╰╴Failed to open directory: .......

[SUPERCILEX]

if multiple rmz on same folder, everything fails with

That's expected since you're trying to delete the same thing twice. It sucks UX wise but is hard to fix correctly while retaining parallel deletion: consider a directory structure a/b/c with an attempt to delete a and a/b (note that a/b could be under a different name if a is symlinked in the second case). If you've started deleting one of those, you'll be screwed trying to delete the other. This could be fixed by first canonicalizing all the paths, pruning parent/child relationships, and dropping duplicates, and only then beginning execution. Not sure that's really in the spirit of this crate though. Obviously the performance cost is irrelevant for like 10 things, but if you're passing in say 1000 paths I'm going to start getting grumpy.

---

Thoughts on a NO_UNSHARE envvar that doesn't call the syscall at all? I'd be fine throwing that in as an undocumented feature for the default (but bad IMO) docker configs.

[jhheider]

not my issue, but wouldn't, from my reading of the help, --force work on non-existent arguments? or does that only work at the parsing/planning stage, and not if you later encounter a missing file.

many tools will freak out if you try to double-remove something, so that seems like normalish behavior.

NO_UNSHARE seems like a reasonable solution. You could probably throw it in a FAQ that tells people why docker isn't doing the right thing. :)

[SUPERCILEX]

does that only work at the parsing/planning stage, and not if you later encounter a missing file.

Correct, since nothing prevents you from creating/deleting files in a directory we're trying to delete, so trying to make it work in those cases is somewhat pointless IMO.

You could probably throw it in a FAQ that tells people why docker isn't doing the right thing. :)

Nah, cuz then it's an official feature and I want to be able to change my mind later. :) This issue should pop up during a search so I think it's fine.

[jhheider]

Nah, cuz then it's an official feature and I want to be able to change my mind later. :)

respect. :)