get-env.sh interprets hex literals in function bodies

[winterqt]

Describe the bug

Hex literals such as \xAB in functions are interpreted via get-env.sh's usage of declare to get the functions to output. As of https://github.com/NixOS/nixpkgs/commit/488395c0f80e2863e9cf83ffb029a330c8e19573, a function that uses these now exist in stdenv, making this an issue when outputting JSON using print-dev-env.

error: [json.exception.parse_error.101] parse error at line 57, column 178: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '" \n    local fn=\"$1\";\n    local fd;\n    local magic;\n    exec {fd}< \"$fn\";\n    read -r -n 4 -u \"$fd\" magic;\n    exec {fd}>&-;\n    if [[ \"$magic\" = ''

Steps To Reproduce

1. Run nix print-dev-env with an installable that uses Nixpkgs commit 488395c0 or newer

Expected behavior

For the hex literals to be properly escaped, creating valid JSON.

nix-env --version output

nix-env (Nix) 2.4pre20210908_3c56f62

Additional context

https://github.com/nix-community/nix-direnv/issues/113, https://github.com/nix-community/nix-direnv/issues/113#issuecomment-921225679

[winterqt]

@edolstra Do you have any suggestions on how to fix this? I’d definitely be open to PRing a fix, but unsure how best to tackle this, since it seems like declare just... doesn’t support not interpreting the hex literal escape code.

[edolstra]

Probably the easiest solution is to revert NixOS/nixpkgs@488395c or change it to not use \x.

[winterqt]

Probably the easiest solution is to revert NixOS/nixpkgs@488395c

Agreed (see https://github.com/NixOS/nixpkgs/pull/138186), since I'm not sure if there's an easy way this can be changed to not use \x.

Nevermind.

[kira-bruneau]

It's pretty strange that declare -f just prints the raw bytes. It looks like it's pretty printing the AST, and not escaping strings that were unescaped during parsing.

This could be fixed in get-env.sh by modifying __escapeString to escape any characters that fall outside of 0x20..0x7F with \u, but I'm not sure of an easy way to do this with just POSIX regexps. I think the only option is to escape each character by manually iterating over the string.

[kira-bruneau]

This is harder to fix than I thought. I tried to shove the non-printable characters into unicode code points, but I forgot that everything from \u007f to \u00ff doesn't cleanly map back into a byte when translated to UTF-8 :sweat:

[edolstra]

I think the easiest solution is to change

$'\xfe\xed\xfa\xcf'

to

$(printf '\xfe\xed\xfa\xcf')

[winterqt]

@edolstra A similar fix was merged into Nixpkgs: https://github.com/NixOS/nixpkgs/pull/138334.

Should this issue be closed, or should we keep it open since this is still technically an issue?

[edolstra]

Let's keep it open.

[solomon-b]

I'm running into this issue on 21.05 with nixUnstable and am unable to run nix flake update. I tried using the nix overlay from its repo but the build failed. Is there some other suggested method to resolve this issue on 21.05?

edit: I tried cloning and building the 2.4 tag and the 2.4-maintainenance branch of nix and they both still get the same error when i call nix flake update. Is there a fixed version somewhere?

➜ ../nix/result/bin/nix --version
nix (Nix) 2.4

➜ ../nix/result/bin/nix flake update
warning: Git tree '/home/solomon/Development/Nix/nixos-config' is dirty
error: [json.exception.parse_error.101] parse error at line 5, column 1: syntax error while parsing object key - invalid literal; last read: '"locked": {<U+000A><'; expected string literal

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-12-09-nix-team-meeting-minutes-15/23951/1