Open aanderse opened 2 years ago
yes, its a bug in nix develop
, works with nix-shell
cd /tmp
mkdir "yes space"
cd "yes space"
cat >flake.nix <<EOF
{
inputs.nixpkgs.url = "github:NixOS/nixpkgs?rev=1829c5b002d128c9c94f43d8b11c0added3863eb";
outputs = { self, nixpkgs }:
let pkgs = import nixpkgs { system = "x86_64-linux"; }; in
{ devShell.x86_64-linux = pkgs.mkShell { }; };
}
EOF
echo 'int main() { return 0; }' >main.cc
nix develop
g++ main.cc
# ld: cannot find space/outputs/out/lib64: No such file or directory
env | grep outputs/out/lib64
# NIX_LDFLAGS=-rpath /tmp/yes space/outputs/out/lib64 -rpath /tmp/yes space/outputs/out/lib
echo $NIX_LDFLAGS
# -rpath /tmp/yes space/outputs/out/lib64 -rpath /tmp/yes space/outputs/out/lib
exit
nix-shell -E 'with import <nixpkgs> {}; pkgs.mkShell {}'
g++ main.cc && echo ok
# ok
echo $NIX_LDFLAGS
# -rpath /nix/store/rwzyc7anbdhw4cjh45r4m172xirncp8h-nix-shell/lib64 -rpath /nix/store/rwzyc7anbdhw4cjh45r4m172xirncp8h-nix-shell/lib
edit: added NIX_LDFLAGS
_addRpathPrefix
assumes that $1
is a nix store path = no need to escape or quote
# Add $1/lib* into rpaths.
# The function is used in multiple-outputs.sh hook,
# so it is defined here but tried after the hook.
_addRpathPrefix() {
if [ "${NIX_NO_SELF_RPATH:-0}" != 1 ]; then
export NIX_LDFLAGS="-rpath $1/lib ${NIX_LDFLAGS-}"
if [ -n "${NIX_LIB64_IN_SELF_RPATH:-}" ]; then
export NIX_LDFLAGS="-rpath $1/lib64 ${NIX_LDFLAGS-}"
fi
if [ -n "${NIX_LIB32_IN_SELF_RPATH:-}" ]; then
export NIX_LDFLAGS="-rpath $1/lib32 ${NIX_LDFLAGS-}"
fi
fi
}
also called from pkgs/build-support/setup-hooks/multiple-outputs.sh
possible solution
https://stackoverflow.com/questions/70822095/bash-equivalent-of-pythons-shlex-quote
_addRpathPrefix() {
quoted_path="${1@Q}"
echo "quoted_path=$quoted_path"
escaped_path="$(printf '%q' "$1")"
echo "escaped_path=$escaped_path"
}
_addRpathPrefix "/tmp/yes space"
# quoted_path='/tmp/yes space'
# escaped_path=/tmp/yes\ space
_addRpathPrefix "/tmp/no-space"
# quoted_path='/tmp/no-space'
# escaped_path=/tmp/no-space
ping @vcunat @thufschmitt @trofi
I think that should be fixed right in stdenv
with one of the solutions above. As far as I can tell, all the other occurrences of $out
are properly quoted, so let's fix that one too.
Fwiw, it is technically necessary to quote store paths because the store directory itself could be set to something with spaces in it
is this
export NIX_LDFLAGS="-rpath $(escapeString "$1/lib") ${NIX_LDFLAGS-}"
so much prettier than this
export NIX_LDFLAGS="-rpath $(printf '%q' "$1/lib") ${NIX_LDFLAGS-}"
to justify adding a library function escapeString
?
probably no
we need to escape strings in a few other places too
Spaces in PATH
(or any paths) are hard. It would be nice to support it in more places if easily feasible. I don't think quoting achieves the intended goal (but I did not test heavily, i provide a silly broken example scenario below).
In practice I see a few obstacles:
nixpkgs
for $out
it looks like only 5K instances quote it while 25K don't. Including nixpkgs
manual itself.glibc
's LD_PRELOAD
(as an example) considers whitespace a separator (along with :
). I would not expect paths with whitespace (escaped or not) to work consistently.I think that adding escapes will fix some use sites and will break other use sites. It depends so much on the usage context:
$ mkdir 'a b'
$ cmd="ls -ld /"
$ $cmd
drwxr-xr-x 1 root root 108 Aug 15 14:27 /
$ cmd="ls -ld $(printf '%q' 'a b')"
$ $cmd
ls: cannot access 'a\': No such file or directory
ls: cannot access 'b': No such file or directory
Is it an unreasonable usage? Maybe. Bash arrays would be less prone to such issues. And it's also very common one, including outside world like ./configure
code.
But my worry should not stop you from fixing "obviously correct"(c) cases. I suggest raising PRs for specific cases and see if you can make it to work.
another possible solution:
urlencode /tmp/yes space
as file:///tmp/yes%20space
this makes for-loops easier
I don't think quoting achieves the intended goal
true. this is easy to test:
where is NIX_LDFLAGS
parsed?
pkgs/build-support/cc-wrapper/cc-wrapper.sh
for i in $(filterRpathFlags "$linkType" $NIX_LDFLAGS_@suffixSalt@); do
if [ "${i:0:3}" = -L/ ]; then
extraAfter+=("$i")
else
extraAfter+=("-Wl,$i")
fi
done
could be changed to
for i in $(filterRpathFlags "$linkType" $NIX_LDFLAGS_@suffixSalt@); do
if [ "${i:0:3}" = -L/ ]; then
extraAfter+=("$i")
elif [ "${i:0:10}" = -Lfile:/// ]; then
extraAfter+=("-L$(urldecodeString "${i:9}")")
elif [ "${i:0:8}" = file:/// ]; then
extraAfter+=("-Wl,$(urldecodeString "${i:7}")")
else
extraAfter+=("-Wl,$i")
fi
done
i patched cc-wrapper.sh
= $(which gcc)
with my writable-nix-store
so now this works
NIX_LDFLAGS="-rpath file:///tmp/yes%20space/outputs/out/lib" gcc main.c
where is NIX_LDFLAGS
modified?
problem with file URL encoding: works only for file paths = is too specific
we should use a more generic solution, so back to
s="a b"
echo "$(printf '%q' "$s")" # escape
# a\ b
s="a b"
echo "${s@Q}" # quote, also when not needed
# 'a b'
# parse
echo "\"a a\" b\\ b 'c c'" | xargs printf 'arg: %s\n'
# arg: a a
# arg: b b
# arg: c c
# limitation: xargs breaks on "\"" and '\''
# but $(printf '%q' "$s") and ${s@Q} still work
echo "\"a \\\" a\"" | xargs printf 'arg: %s\n'
# xargs: unmatched double quote
s="\"a \\\" a\""; echo "${s@Q} $(printf '%q' "$s")" | xargs printf 'arg: %s\n'
# arg: "a \" a"
# arg: "a \" a"
echo "'c \\' c'" | xargs printf 'arg: %s\n'
# xargs: unmatched single quote
s="'c \\' c'"; echo "${s@Q} $(printf '%q' "$s")" | xargs printf 'arg: %s\n'
# arg: 'c \' c'
# arg: 'c \' c'
# parse to array
readArrayString() {
readarray -d '' "$1" < <(<<<"$2" xargs printf '%s\0')
}
x='1 1'
y='2 2'
s="$(printf '%q' "$x") ${y@Q}"
readArrayString a "$s"
[[ "${a[0]}" == '1 1' ]]
[[ "${a[1]}" == '2 2' ]]
this also solves the edge-case:
pass whitespace in strings to array variables like configureFlags
.
in nix, these are beautiful arrays
{
configureFlags = [
"a a"
"b b"
];
}
but these strings are "overparsed" to
./configure a a b b
expected result:
./configure "a a" "b b"
workaround to avoid radical changes to the compiler toolchains:
create a link (hardlink or symlink) to the workdir of nix develop
.
that link has no whitespace, so we can use it for $out
example:
/home/user/.cache/nix-develop/xxx
→ /tmp/yes space
now we have a simple rpath like
NIX_LDFLAGS=-rpath /home/user/.cache/nix-develop/xxx/outputs/out/lib
hardlinks pro: allow to move workdir within the same filesystem con: must be on the same filesystem
I think it is fixed:
$ nix --version
nix (Nix) 2.8.1
$ nix develop
error: 'file:///home/Et7f3/programanodin/yes space' is not a valid URL
? Or you can update the scenario.
I guess not, I get a similar issue with rust (cargo) when building a subproject which resides inside a git repo with spaces in its name. This is the ouput of the NIX_LDFLAGS variable:
-rpath /home/tom/Desktop/Studium/Information Systems Engineering/Mathematische Algorithmen/Praktika/01/outputs/out/lib64 -rpath /home/tom/Desktop/Studium/Information Systems Engineering/Mathematische Algorithmen/Praktika/01/outputs/out/lib -rpath '/home/tom/Desktop/Studium/Information Systems Engineering/Mathematische Algorithmen/Praktika/01/outputs/out/lib' -L/nix/store/w9saki3n1lnm8wlx2hcsv6114q3js9d8-rust-nightly-complete-with-std-2023-03-31/lib -L/nix/store/xh8mdn84clglsa9wygn5a23b5s767fb1-clippy-nightly-complete-with-std-2023-03-31/lib -L/nix/store/w9saki3n1lnm8wlx2hcsv6114q3js9d8-rust-nightly-complete-with-std-2023-03-31/lib -L/nix/store/xh8mdn84clglsa9wygn5a23b5s767fb1-clippy-nightly-complete-with-std-2023-03-31/lib
I guess not, I get a similar issue with rust (cargo) when building a subproject which resides inside a git repo with spaces in its name. This is the ouput of the NIX_LDFLAGS variable:
-rpath /home/tom/Desktop/Studium/Information Systems Engineering/Mathematische Algorithmen/Praktika/01/outputs/out/lib64 -rpath /home/tom/Desktop/Studium/Information Systems Engineering/Mathematische Algorithmen/Praktika/01/outputs/out/lib -rpath '/home/tom/Desktop/Studium/Information Systems Engineering/Mathematische Algorithmen/Praktika/01/outputs/out/lib' -L/nix/store/w9saki3n1lnm8wlx2hcsv6114q3js9d8-rust-nightly-complete-with-std-2023-03-31/lib -L/nix/store/xh8mdn84clglsa9wygn5a23b5s767fb1-clippy-nightly-complete-with-std-2023-03-31/lib -L/nix/store/w9saki3n1lnm8wlx2hcsv6114q3js9d8-rust-nightly-complete-with-std-2023-03-31/lib -L/nix/store/xh8mdn84clglsa9wygn5a23b5s767fb1-clippy-nightly-complete-with-std-2023-03-31/lib
Can you share a minimal repro so I can help fixing ?
Sure thing, thank you.
So here is the repo https://github.com/MordragT/nix-ld-flags
When I try to build the project with nix build
an error message shows,
which pretty much explains why it isn't working:
error: store path 'pbb8kcjyaw50p10pdpi0y550wfwj45xg-Mathematische Algorithmen' contains illegal character ' '
I don't know why in my other repository this is working, I will have to investigate that.
But nix develop
is working flawlessly.
When trying to build the project with cargo build
the linking fails, as the ' ' is used for separation:
/nix/store/8qm6sjqa09a03glzmafprpp69k74l4lm-binutils-2.40/bin/ld: cannot find Systems: No such file or directory
/nix/store/8qm6sjqa09a03glzmafprpp69k74l4lm-binutils-2.40/bin/ld: cannot find Engineering/Mathematische: No such file or directory
/nix/store/8qm6sjqa09a03glzmafprpp69k74l4lm-binutils-2.40/bin/ld: cannot find Algorithmen/outputs/out/lib64: No such file or directory
/nix/store/8qm6sjqa09a03glzmafprpp69k74l4lm-binutils-2.40/bin/ld: cannot find Systems: No such file or directory
/nix/store/8qm6sjqa09a03glzmafprpp69k74l4lm-binutils-2.40/bin/ld: cannot find Engineering/Mathematische: No such file or directory
/nix/store/8qm6sjqa09a03glzmafprpp69k74l4lm-binutils-2.40/bin/ld: cannot find Algorithmen/outputs/out/lib: No such file or directory
just got bitten by this
@MordragT did you ever manage to work around this?
@MordragT did you ever manage to work around this?
No I just renamed my paths and removed all the spaces
Someone should use o1
from openai and go through stdenv and tell it to fix quoting.
Describe the bug
A clear and concise description of what the bug is.
I am using
mkShell
andc++
. I can't link my code when the code resides in a folder which has a space in the name. Maybe I'm missing something obvious...Steps To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
I want to be able to compile
c++
code.