wyoming-satellite service fails to start

miraz12 commented 2 weeks ago

Describe the bug

When trying to run wyoming-satellite service it fails to start due to .wyoming-satellite-wrapped reporting syntax error near unexpected token lambda in the following way:

Sep 23 10:21:13 nixpi4 systemd-timesyncd[688]: Network configuration changed, trying to establish connection.
Sep 23 10:21:13 nixpi4 systemd-timesyncd[688]: Contacted time server 85.24.237.71:123 (2.nixos.pool.ntp.org).
Sep 23 10:21:13 nixpi4 wyoming-satellite-start[1969]: /nix/store/h8x240qrr70l1ijilah09wk1c1a7manz-wyoming-satellite-1.2.0/bin/.wyoming-satellite-wrapped: /nix/store/h8x240qrr70l1ijilah09wk1c1a7manz-wyoming-satellite-1.2.0/bin/wyoming-satellite: line 3: syntax error near unexpected token `lambda'
Sep 23 10:21:13 nixpi4 wyoming-satellite-start[1969]: /nix/store/h8x240qrr70l1ijilah09wk1c1a7manz-wyoming-satellite-1.2.0/bin/.wyoming-satellite-wrapped: /nix/store/h8x240qrr70l1ijilah09wk1c1a7manz-wyoming-satellite-1.2.0/bin/wyoming-satellite: line 3: `import sys;import site;import functools;sys.argv[0] = '/nix/store/h8x240qrr70l1ijilah09wk1c1a7manz-wyoming-satellite-1.2.0/bin/wyoming-satellite';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/h8x240qrr70l1ijilah09wk1c1a7manz-wyoming-satellite-1.2.0/lib/python3.12/site-packages','/nix/store/x8f1kjmiqchgjgb7y3bqdv5v91739a73-python3.12-pyring-buffer-1.0.0/lib/python3.12/site-packages','/nix/store/5hdmy5m19ipg4nkvp4wn8vkcal3rp65h-python3.12-wyoming-1.5.4/lib/python3.12/site-packages','/nix/store/m6dff33fbs89718rvp1lyiv7brhjqpnz-python3.12-zeroconf-0.133.0/lib/python3.12/site-packages','/nix/store/c9dlc971dzbd7n1m0xpjgbivaj9wkh2y-python3.12-ifaddr-0.2.0/lib/python3.12/site-packages','/nix/store/1zqq5dlb2d6z3l4d7yfpi5bdyxnqpy8n-python3.12-webrtc-noise-gain-1.2.4/lib/python3.12/site-packages'], site._init_pathinfo());'

Steps To Reproduce

Setup service with unstable pkg version (main one is broken)

wyoming.satellite = {
enable = true;
user = "root";
name = "Living-room";
vad.enable = false;
package = pkgs-unstable.wyoming-satellite;
};

Rebuild and check logs

Expected behavior

The service starts wyoming-satellite successfully.

Additional context

Manually running the file with python works without problem so it seems like the service is the issue.

I would be open to look at it myself more but I'm unsure in which end do pull.

Notify maintainers

@mweinelt

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 6.6.52, NixOS, 24.05 (Uakari), 24.05.20240920.944b2ae`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.5`
 - channels(root): `"nixos-24.05.4639.6f6c45b5134a"`
 - nixpkgs: `/nix/store/mvz96grv31nxq69ldw5a5pn2qh1s6ca6-source`

Add a :+1: reaction to issues you find important.

mweinelt commented 2 weeks ago

Can you post the contents of the file referenced under ExecStart= on your system? And if that calls another file, please follow the trail until you arrive at the wrapper.

miraz12 commented 2 weeks ago

Sure thing!

wyoming-satellite.service:

```ini
ExecStart=/nix/store/0z389qiacin2ar52dha11fdq9127n6vi-unit-script-wyoming-satellite-start/bin/wyoming-satellite-start

wyoming-satellite-start:

#!/nix/store/rmbznwsk8pznvd5fwf5sllla0k409wa1-bash-5.2p32/bin/bash
set -e
export XDG_RUNTIME_DIR=/run/user/$UID
'/nix/store/h8x240qrr70l1ijilah09wk1c1a7manz-wyoming-satellite-1.2.0/bin/wyoming-satellite' '--uri' 'tcp://0.0.0.0:10700' '--name' 'Living-room' '--mic-command' 'arecord -r 16000 -c 1 -f S16_LE -t raw' '--mic-auto-gain' '5' '--mic-noise-suppression' '2' '--snd-command' 'aplay -r 22050 -c 1 -f S16_LE -t raw'

wyoming-satellite:

#! /nix/store/izpf49b74i15pcr9708s3xdwyqs4jxwl-bash-5.2p32/bin/bash -e
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/h8x240qrr70l1ijilah09wk1c1a7manz-wyoming-satellite-1.2.0/bin'':'/':'}
PATH='/nix/store/h8x240qrr70l1ijilah09wk1c1a7manz-wyoming-satellite-1.2.0/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin'':'/':'}
PATH='/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
export PYTHONNOUSERSITE='true'
exec -a "$0" "/nix/store/h8x240qrr70l1ijilah09wk1c1a7manz-wyoming-satellite-1.2.0/bin/.wyoming-satellite-wrapped"  "$@"

.wyoming-satellite-wrapper:

#!/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin/python3.12
# -*- coding: utf-8 -*-
import sys;import site;import functools;sys.argv[0] = '/nix/store/pdbrj0j34g9v9a8ldp3ff05fsx256nsh-wyoming-satellite-1.2.0/bin/wyoming-satellite';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/pdbrj0j34g9v9a8ldp3ff05fsx256nsh-wyoming-satellite-1.2.0/lib/python3.12/site-packages','/nix/store/x8f1kjmiqchgjgb7y3bqdv5v91739a73-python3.12-pyring-buffer-1.0.0/lib/python3.12/site-packages','/nix/store/5hdmy5m19ipg4nkvp4wn8vkcal3rp65h-python3.12-wyoming-1.5.4/lib/python3.12/site-packages','/nix/store/m6dff33fbs89718rvp1lyiv7brhjqpnz-python3.12-zeroconf-0.133.0/lib/python3.12/site-packages','/nix/store/c9dlc971dzbd7n1m0xpjgbivaj9wkh2y-python3.12-ifaddr-0.2.0/lib/python3.12/site-packages','/nix/store/1zqq5dlb2d6z3l4d7yfpi5bdyxnqpy8n-python3.12-webrtc-noise-gain-1.2.4/lib/python3.12/site-packages','/nix/store/kzwfg91i9lqr5739swhzvwd6xkx385k1-python3.12-pysilero-vad-2.0.0/lib/python3.12/site-packages','/nix/store/fvxlmgcjanv8j0qxzxxgxigq1344zn39-python3.12-numpy-1.26.4/lib/python3.12/site-packages','/nix/store/80wsx8msqh3jv1p70jrpjbdlh1a7pv2s-python3.12-onnxruntime-1.18.1/lib/python3.12/site-packages','/nix/store/5b1xm6jh9hvy3zzvmfskmb3qb2q56vli-python3.12-coloredlogs-15.0.1/lib/python3.12/site-packages','/nix/store/dl1xvp74i1rbd8m869a1vjai1c54yci1-python3.12-humanfriendly-10.0/lib/python3.12/site-packages','/nix/store/zxl97260xfw4pd13jrnkaghkalbin3y5-python3.12-packaging-24.1/lib/python3.12/site-packages'], site._init_pathinfo());
import re
import sys
from wyoming_satellite import __main__
if __name__ == "__main__":
    sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
    sys.exit(__main__.run())

mweinelt commented 2 weeks ago

I somewhat expect two more files.

miraz12 commented 2 weeks ago

The two I added now or some other files?

mweinelt commented 2 weeks ago

That lambda doesn't immediately stand out to me. Can you try nix-store --verify --repair --check-contents?

miraz12 commented 2 weeks ago

[user@system:~]$ nix-store --verify --repair --check-contents
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...

Seems to have run without problem? Do I need to rebuild after too?

mweinelt commented 2 weeks ago

Looks fine to me. No rebuild required. But that doesn't explain the errors you are seeing.

miraz12 commented 2 weeks ago

Alright! This is on my rpi4 but I did try running it on my laptop too with the same issue.

mweinelt commented 2 weeks ago

The diff between your wrapper (green) and mine (red) looks quite innocent:


1c1
< #!/nix/store/pgb120fb7srbh418v4i2a70aq1w9dawd-python3-3.12.5/bin/python3.12
---
> #!/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin/python3.12
8c8
<     "/nix/store/iaf9w5lcj7y4ja7sv0ps4662y0x89hb3-wyoming-satellite-1.2.0/bin/wyoming-satellite"
---
>     "/nix/store/pdbrj0j34g9v9a8ldp3ff05fsx256nsh-wyoming-satellite-1.2.0/bin/wyoming-satellite"
13,24c13,24
<         "/nix/store/iaf9w5lcj7y4ja7sv0ps4662y0x89hb3-wyoming-satellite-1.2.0/lib/python3.12/site-packages",
<         "/nix/store/8hs9yi1hm8c4ljdsfbxfgn2g4jq21xvb-python3.12-pyring-buffer-1.0.0/lib/python3.12/site-packages",
<         "/nix/store/x2plm0b4x70942i6gx28w2049rd61kd7-python3.12-wyoming-1.5.4/lib/python3.12/site-packages",
<         "/nix/store/gz7yqn19qw107h75zjpsr08ygp28kfrh-python3.12-zeroconf-0.132.2/lib/python3.12/site-packages",
<         "/nix/store/z1p20b8z0gxgvd5zh7c4ynm602ma5na6-python3.12-ifaddr-0.2.0/lib/python3.12/site-packages",
<         "/nix/store/nam03flx5c84n6xbja6crrziwn8ypyq5-python3.12-webrtc-noise-gain-1.2.4/lib/python3.12/site-packages",
<         "/nix/store/lz55m43vsn2lggnpf8q7s2w0s4b8d0ra-python3.12-pysilero-vad-1.0.0/lib/python3.12/site-packages",
<         "/nix/store/wigx5inak1i9556y67bh4n43x7677a7a-python3.12-numpy-1.26.4/lib/python3.12/site-packages",
<         "/nix/store/kx27k2c32k7bvaqzai4cdgw10y41pyny-python3.12-onnxruntime-1.18.1/lib/python3.12/site-packages",
<         "/nix/store/jg14vv6923rcnkagsy7vlcpv2k2lysd5-python3.12-coloredlogs-15.0.1/lib/python3.12/site-packages",
<         "/nix/store/nqsxjp72316rgy8p39ifsahj75g9zh87-python3.12-humanfriendly-10.0/lib/python3.12/site-packages",
<         "/nix/store/32q6cvqyhfk7jxhpk2csr8xfxcd20dhw-python3.12-packaging-24.1/lib/python3.12/site-packages",
---
>         "/nix/store/pdbrj0j34g9v9a8ldp3ff05fsx256nsh-wyoming-satellite-1.2.0/lib/python3.12/site-packages",
>         "/nix/store/x8f1kjmiqchgjgb7y3bqdv5v91739a73-python3.12-pyring-buffer-1.0.0/lib/python3.12/site-packages",
>         "/nix/store/5hdmy5m19ipg4nkvp4wn8vkcal3rp65h-python3.12-wyoming-1.5.4/lib/python3.12/site-packages",
>         "/nix/store/m6dff33fbs89718rvp1lyiv7brhjqpnz-python3.12-zeroconf-0.133.0/lib/python3.12/site-packages",
>         "/nix/store/c9dlc971dzbd7n1m0xpjgbivaj9wkh2y-python3.12-ifaddr-0.2.0/lib/python3.12/site-packages",
>         "/nix/store/1zqq5dlb2d6z3l4d7yfpi5bdyxnqpy8n-python3.12-webrtc-noise-gain-1.2.4/lib/python3.12/site-packages",
>         "/nix/store/kzwfg91i9lqr5739swhzvwd6xkx385k1-python3.12-pysilero-vad-2.0.0/lib/python3.12/site-packages",
>         "/nix/store/fvxlmgcjanv8j0qxzxxgxigq1344zn39-python3.12-numpy-1.26.4/lib/python3.12/site-packages",
>         "/nix/store/80wsx8msqh3jv1p70jrpjbdlh1a7pv2s-python3.12-onnxruntime-1.18.1/lib/python3.12/site-packages",
>         "/nix/store/5b1xm6jh9hvy3zzvmfskmb3qb2q56vli-python3.12-coloredlogs-15.0.1/lib/python3.12/site-packages",
>         "/nix/store/dl1xvp74i1rbd8m869a1vjai1c54yci1-python3.12-humanfriendly-10.0/lib/python3.12/site-packages",
>         "/nix/store/zxl97260xfw4pd13jrnkaghkalbin3y5-python3.12-packaging-24.1/lib/python3.12/site-packages",

mweinelt commented 2 weeks ago

line 3: syntax error near unexpected token `lambda'

Oh wait. This line indicates the wrapper is executed through bash, while it clearly has a python shebang?!

miraz12 commented 2 weeks ago

Ahaa good catch! Running the wrapper with python actually works too! But why is the shebang ignored?

mweinelt commented 2 weeks ago

Does the python at /nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin/python3.12 work for you on the arm box? What does file /nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin/python3.12 say?

miraz12 commented 2 weeks ago

/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin/python3.12 Is present at least

miraz12 commented 2 weeks ago

/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin/python3.12: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/ld-linux-x86-64.so.2, BuildID[sha1]=97e0266da2544eb8b3ce9e607323ae7b162ab8b7, for GNU/Linux 3.10.0, not stripped

I cant manually run it with /nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin/python3.12 though but perhaps that is expected?

bash: /nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin/python3.12: cannot execute binary file: Exec format error

mweinelt commented 2 weeks ago

Yeah, so what is happening is that it ignores the shebang and falls back to bash, because it can't execute the binary.

It looks like bad cross, or no system set in a nixpkgs import.

miraz12 commented 2 weeks ago

Ahaa yes that is exactly it! I am deploying with nixops but when I import nixpks-unstable I specify x86_64-linuxfor my other systems and of course it does not work. Made a separate import for aarch64-linux now and it runs as it should!

Thank you for the quick help and sorry for the noise!

NixOS / nixpkgs