luarocks-packages-updater runs into rate limits

flokli commented 1 week ago

Describe the bug

I tried adding a new lua package, by adding it to maintainers/scripts/luarocks-packages.csv. Afterwards I ran nix-shell -p luarocks-packages-updater --run luarocks-packages-updater as instructed in doc/languages-frameworks/lua.section.md and got a wall of errors:

       {"message":"API rate limit exceeded for 88.196.79.174. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}
Error: command exited with exit status: 1
Call to nurl --indent 2 https://github.com/mwild1/luadbi v0.7.3 failed with status: nil

Error: LuaRocks dev bug (please report at https://github.com/luarocks/luarocks/issues).
Arch.: linux-x86_64
...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:350: attempt to concatenate local 'src_str' (a nil value)
stack traceback:
    ...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:350: in function 'convert_spec2nix'
    ...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:544: in function <...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:483>
    (...tail calls...)
    [C]: in function 'xpcall'
    ...nix-0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd.lua:672: in function 'run_command'
    ...luarocks-nix-0-unstable-2024-04-29/bin/.luarocks-wrapped:39: in main chunk
    [C]: in ?
Error: https://api.github.com/repos/leafo/moonscript/commits?per_page=1: status code 403
Call to nurl --indent 2 https://github.com/leafo/moonscript.git failed with status: nil

Error: LuaRocks dev bug (please report at https://github.com/luarocks/luarocks/issues).
Arch.: linux-x86_64
...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:350: attempt to concatenate local 'src_str' (a nil value)
stack traceback:
    ...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:350: in function 'convert_spec2nix'
    ...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:544: in function <...0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd/nix.lua:483>
    (...tail calls...)
    [C]: in function 'xpcall'
    ...nix-0-unstable-2024-04-29/share/lua/5.2/luarocks/cmd.lua:672: in function 'run_command'
    ...luarocks-nix-0-unstable-2024-04-29/bin/.luarocks-wrapped:39: in main chunk
    [C]: in ?
Traceback (most recent call last):
  File "/nix/store/qr6v4x47c7cxlma95pn8zqnl02qzj6l9-luarocks-packages-updater-0.0.1/bin/.luarocks-packages-updater-wrapped", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/nix/store/qr6v4x47c7cxlma95pn8zqnl02qzj6l9-luarocks-packages-updater-0.0.1/lib/python3.12/site-packages/updater.py", line 216, in main
    editor.run()
  File "/nix/store/13f71n783zs7d9ac63bkdazx8kr14dd0-pluginupdate-py/pluginupdate.py", line 568, in run
    getattr(self, command)(args)
  File "/nix/store/qr6v4x47c7cxlma95pn8zqnl02qzj6l9-luarocks-packages-updater-0.0.1/lib/python3.12/site-packages/updater.py", line 99, in update
    update_plugins(self, args)
  File "/nix/store/13f71n783zs7d9ac63bkdazx8kr14dd0-pluginupdate-py/pluginupdate.py", line 803, in update_plugins
    redirects = update()
                ^^^^^^^^
  File "/nix/store/qr6v4x47c7cxlma95pn8zqnl02qzj6l9-luarocks-packages-updater-0.0.1/lib/python3.12/site-packages/updater.py", line 136, in update
    results = pool.map(_prefetch, sorted_plugin_specs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
           ^^^^^^^^^^^^^^^^
  File "/nix/store/qr6v4x47c7cxlma95pn8zqnl02qzj6l9-luarocks-packages-updater-0.0.1/lib/python3.12/site-packages/updater.py", line 202, in generate_pkg_nix
    output = subprocess.check_output(cmd, text=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['luarocks', 'nix', '--maintainers=vyp', 'lpeg']' returned non-zero exit status 99.

Apparently Nix ran into github rate limits. We should probably improve error handling in the updater (so it doesn't invoke luarocks in case of failure), as well as document how to prevent running into these rate limits in the docs.

Steps To Reproduce

Steps to reproduce the behavior:

nix-shell -p luarocks-packages-updater --run luarocks-packages-updater

Notify maintainers

@teto

Add a :+1: reaction to issues you find important.

flokli commented 1 week ago

I got further by running the updater script with `NIX_CURL_FLAG="--netrc-file = /path/to/a/netrc" which contains a scoped personal access token for public repos.

It looks like this is picked up by nix flake prefetch.

However there still seem to be some issues with error handling and/or faulty/broken/regressed lines in the .csv, causing the process to be quite painful.

flokli commented 1 week ago

Hmmh, no, I got another message about being rate limited, so I guess the NIX_CURL_FLAG env var does not apply here.

flokli commented 1 week ago

It seems luarocks-nix is doing the downloading here:

https://github.com/nix-community/luarocks-nix/blob/9d0440da358eac11afdbef392e2cf3272a8c7101/src/luarocks/fetch.lua#L109

(and that's where it's failing now). How do I configure this to use my PAT?

teto commented 1 week ago

there is a warning that pops up when you launch the updater without github-token

$ luarocks-packages-updater --help
....
  --github-token GITHUB_TOKEN, -t GITHUB_TOKEN
                        Allows to set --proc to higher values. Uses GITHUB_API_TOKEN environment variables as the default value.

--debug=DEBUG also shows more of the failures Any improvement welcome. Out of curiosity what package are you trying to add ?

NixOS / nixpkgs