search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.39k stars 13.61k forks source link

`luarocks-packages-updater` script is flaky #339236

Open GaetanLepage opened 1 week ago

GaetanLepage commented 1 week ago

Describe the bug

The luarocks-packages-updater script has a very flaky behavior and crashes. Because of that, I have not been able to update the luaPackages package set for a while and it starts to affect the vimPlugins set as well.

Logs:

``` Traceback (most recent call last): File "/nix/store/pm87llw5n1j6ygfkzs5zcvzjwgyb988r-luarocks-packages-updater-0.1/bin/..luarocks-packages-updater-wrapped-wrapped", line 222, in main() File "/nix/store/pm87llw5n1j6ygfkzs5zcvzjwgyb988r-luarocks-packages-updater-0.1/bin/..luarocks-packages-updater-wrapped-wrapped", line 218, in main editor.run() File "/nix/store/pm87llw5n1j6ygfkzs5zcvzjwgyb988r-luarocks-packages-updater-0.1/lib/pluginupdate.py", line 568, in run getattr(self, command)(args) File "/nix/store/pm87llw5n1j6ygfkzs5zcvzjwgyb988r-luarocks-packages-updater-0.1/bin/..luarocks-packages-updater-wrapped-wrapped", line 101, in update update_plugins(self, args) File "/nix/store/pm87llw5n1j6ygfkzs5zcvzjwgyb988r-luarocks-packages-updater-0.1/lib/pluginupdate.py", line 803, in update_plugins redirects = update() ^^^^^^^^ File "/nix/store/pm87llw5n1j6ygfkzs5zcvzjwgyb988r-luarocks-packages-updater-0.1/bin/..luarocks-packages-updater-wrapped-wrapped", line 138, in update results = pool.map(_prefetch, sorted_plugin_specs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/pgb120fb7srbh418v4i2a70aq1w9dawd-python3-3.12.5/lib/python3.12/multiprocessing/pool.py", line 367, in map return self._map_async(func, iterable, mapstar, chunksize).get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/pgb120fb7srbh418v4i2a70aq1w9dawd-python3-3.12.5/lib/python3.12/multiprocessing/pool.py", line 774, in get raise self._value File "/nix/store/pgb120fb7srbh418v4i2a70aq1w9dawd-python3-3.12.5/lib/python3.12/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) ^^^^^^^^^^^^^^^^^^^ File "/nix/store/pgb120fb7srbh418v4i2a70aq1w9dawd-python3-3.12.5/lib/python3.12/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) ^^^^^^^^^^^^^^^^ File "/nix/store/pm87llw5n1j6ygfkzs5zcvzjwgyb988r-luarocks-packages-updater-0.1/bin/..luarocks-packages-updater-wrapped-wrapped", line 204, in generate_pkg_nix output = subprocess.check_output(cmd, text=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/pgb120fb7srbh418v4i2a70aq1w9dawd-python3-3.12.5/lib/python3.12/subprocess.py", line 466, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/pgb120fb7srbh418v4i2a70aq1w9dawd-python3-3.12.5/lib/python3.12/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['luarocks', 'nix', '--maintainers=vyp', 'lpeg']' returned non-zero exit status 99. ```

Steps To Reproduce

Steps to reproduce the behavior:

  1. nix run .#luarocks-packages-updater (from a clone of nixpkgs)

Expected behavior

The script runs fine and does not crash.

Notify maintainers

cc @mrcjkb @teto

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.47, NixOS, 24.11 (Vicuna), 24.11.20240831.12228ff`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.5`
 - nixpkgs: `/nix/store/bd4fmzws6n5542khxbifbkr6nrygi232-source`

Add a :+1: reaction to issues you find important.

GaetanLepage commented 1 week ago

The package listed in the final stacktrace is not always the same by the way.

Also, earlier during the execution, those error messages appear several times (although they do not directly stop execution):

Error: LuaRocks dev bug (please report at https://github.com/luarocks/luarocks/issues).
Arch.: linux-x86_64
...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:350: attempt to concatenate local 'src_str' (a nil value)
stack traceback:
    ...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:350: in function 'convert_spec2nix'
    ...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:544: in function <...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:483>
    (...tail calls...)
    [C]: in function 'xpcall'
    ...nix-0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd.lua:672: in function 'run_command'
    ...luarocks-nix-0-unstable-2024-04-29/bin/.luarocks-wrapped:39: in main chunk
    [C]: in ?
Error: https://api.github.com/repos/leafo/moonscript/commits?per_page=1: status code 403
Call to nurl --indent 2 https://github.com/leafo/moonscript.git failed with status: nil
teto commented 1 week ago

try running it with --debug=DEBUG and you might see the real error. It runs updates in parallel, which might explain why you dont see the same output everytime

GaetanLepage commented 6 days ago

Thanks, I have done that, along with -p 1. I confirm that the script reliably fails on lpeg. (see the logs in the PR description)

GaetanLepage commented 3 days ago

The issue is caused by lpeg's src: http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz being 404. The script calls luarocks nix --maintainers=vyp which itself calls nix-prefetch-url which fails to download the source and returns null.

I propose to remove the package. Does someone has a better solution ?

GaetanLepage commented 3 days ago

Actually, the prior release is still online. Maybe the author has yanked the latest one but forgot to remove it from luarocks...

mrcjkb commented 3 days ago

The issue is caused by lpeg's src: http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz being 404.

The URL works fine on my end :thinking:

GaetanLepage commented 3 days ago

I actually don't have a 404, but rather a 301, sorry:

✗ nix-prefetch-url http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz
warning: error: unable to download 'http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz': HTTP error 301 (curl error: SSL peer certificate or SSH remote key was not OK); retrying in 296 ms
warning: error: unable to download 'http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz': HTTP error 301 (curl error: SSL peer certificate or SSH remote key was not OK); retrying in 552 ms
warning: error: unable to download 'http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz': HTTP error 301 (curl error: SSL peer certificate or SSH remote key was not OK); retrying in 1092 ms
warning: error: unable to download 'http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz': HTTP error 301 (curl error: SSL peer certificate or SSH remote key was not OK); retrying in 2658 ms
error: unable to download 'http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz': HTTP error 301 (curl error: SSL peer certificate or SSH remote key was not OK)
GaetanLepage commented 3 days ago

My web browser (firefox) doesn't complain, but wget/nix-prefetch-url does.

teto commented 2 days ago

301 is for redirection: 

➜ curl 'http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz'
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz">here</a>.</p>
</body></html>

If I follow the redirection:

curl 'https://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz'
curl: (35) OpenSSL/3.0.14: error:16000069:STORE routines::unregistered scheme

Note sure what that means. certificate issue ? firefox is fine with https://www.inf.puc-rio.br/~roberto/lpeg/ . No big warning about a self-signed certificate (website is let's encrypt).

Several options:

  1. ideally someone asks them to fix it. I've done my fair deal of lua ecosystem fixup so I pass. They should at least
  2. you can modify in the luarocks csv file to use an alternate valid rockspec (towards one of your fork for instance, I've done it in the past for faulty packages. Not great but well..)
  3. remove lpeg from the update list and add it back. Or move it to lua/overrides.nix so it doesn't get autoupdated for now
mrcjkb commented 2 days ago

I'd suggest we go with option 3. (move to overrides) as a temporary hotfix and bug the rockspec maintainer for a fix (I've just sent them an e-mail).

gvvaughan commented 1 day ago

LPeg rockspec maintainer here. i released a new revision of the rockspec earlier, updating all the URLs to https.

GaetanLepage commented 1 day ago

LPeg rockspec maintainer here. i released a new revision of the rockspec earlier, updating all the URLs to https.

Thank you for helping us with this !

Unfortunately, nix-prefetch-url still complains as follows:

✗ nix-prefetch-url https://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz
warning: error: unable to download 'https://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 254 ms
warning: error: unable to download 'https://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 503 ms
warning: error: unable to download 'https://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 1132 ms
warning: error: unable to download 'https://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 2676 ms
error: unable to download 'https://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz': SSL peer certificate or SSH remote key was not OK (60)
gvvaughan commented 22 hours ago

I don't think this can be an issue with the rockspec. However, I have no trouble downloading the lpeg tarball using curl, wget, or clicking the link with my browser from either the http:// or https:// protocols.

$ curl -o lpeg-1.1.0.tar.gz -L 'http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   266  100   266    0     0    391      0 --:--:-- --:--:-- --:--:--   391
100 78042  100 78042    0     0  38035      0  0:00:02  0:00:02 --:--:--  112k

$ ls -l
-rw-rw-r--@ 1 gvv  staff  78042 Sep 14 16:46 lpeg-1.1.0.tar.gz

$ md5sum lpeg-1.1.0.tar.gz 
842a538b403b5639510c9b6fffd2c75b  lpeg-1.1.0.tar.gz

$ rm -rf lpeg*

$ curl -o lpeg-1.1.0.tar.gz -L 'https://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 78042  100 78042    0     0  73715      0  0:00:01  0:00:01 --:--:-- 73763

$ md5sum lpeg-1.1.0.tar.gz
842a538b403b5639510c9b6fffd2c75b  lpeg-1.1.0.tar.gz

$ ls -l
-rw-rw-r--@ 1 gvv  staff  78042 Sep 14 16:49 lpeg-1.1.0.tar.gz

$ luarocks install lpeg
Installing https://luarocks.org/lpeg-1.1.0-2.src.rock

lpeg 1.1.0-1 depends on lua >= 5.1 (5.4-1 provided by VM: success)
env MACOSX_DEPLOYMENT_TARGET=11.0 gcc -O2 -fPIC -I/opt/homebrew/opt/lua/include/lua5.4 -c lpcap.c -o lpcap.o
env MACOSX_DEPLOYMENT_TARGET=11.0 gcc -O2 -fPIC -I/opt/homebrew/opt/lua/include/lua5.4 -c lpcode.c -o lpcode.o
env MACOSX_DEPLOYMENT_TARGET=11.0 gcc -O2 -fPIC -I/opt/homebrew/opt/lua/include/lua5.4 -c lpcset.c -o lpcset.o
env MACOSX_DEPLOYMENT_TARGET=11.0 gcc -O2 -fPIC -I/opt/homebrew/opt/lua/include/lua5.4 -c lpprint.c -o lpprint.o
env MACOSX_DEPLOYMENT_TARGET=11.0 gcc -O2 -fPIC -I/opt/homebrew/opt/lua/include/lua5.4 -c lptree.c -o lptree.o
env MACOSX_DEPLOYMENT_TARGET=11.0 gcc -O2 -fPIC -I/opt/homebrew/opt/lua/include/lua5.4 -c lpvm.c -o lpvm.o
env MACOSX_DEPLOYMENT_TARGET=11.0 gcc  -bundle -undefined dynamic_lookup -all_load -o /var/folders/_v/znckzrz53bd8nnkrzm09cf_80000gn/T/luarocks_build-LPeg-1.1.0-2--Cba67n/lpeg.so lpcap.o lpcode.o lpcset.o lpprint.o lptree.o lpvm.o
No existing manifest. Attempting to rebuild...
lpeg 1.1.0-2 is now installed in /opt/homebrew (license: MIT/X11)

Maybe nix-prefetch-url is tripping over 301's or has a bad root CA or something?