codedownio / julia2nix

Generate Nix expressions for building a Julia depot with a set of packages
53 stars 6 forks source link

julia2nix fails with Julia 1.7 #26

Closed djacu closed 2 years ago

djacu commented 2 years ago

Describe the bug Tried to run julia2nix on a depot created with Julia 1.7 and it fails.

To Reproduce Following the README:

Create the depot directory.

[bakerdn@tweag-laptop:~/tweag/jupyterWith]$ mkdir depot

[bakerdn@tweag-laptop:~/tweag/jupyterWith]$ cd depot/

[bakerdn@tweag-laptop:~/tweag/jupyterWith/depot]$ nix run nixpkgs#julia_17-bin

Pressed ] to enter package mode.

(@v1.7) pkg> activate .
  Activating new project at `~/tweag/jupyterWith/depot`

(depot) pkg> add IJulia
   Resolving package versions...
    Updating `~/tweag/jupyterWith/depot/Project.toml`
  [7073ff75] + IJulia v1.23.3
    Updating `~/tweag/jupyterWith/depot/Manifest.toml`
  [8f4d0f93] + Conda v1.7.0
  [7073ff75] + IJulia v1.23.3
...
Precompiling project...
  5 dependencies successfully precompiled in 3 seconds (10 already precompiled)

Ctrl-D to exit.

Cloned the julia2nix repo.

[bakerdn@tweag-laptop:~/tweag]$ git clone git@github.com:codedownio/julia2nix.git

From the depot directory, run julia2nix.

[bakerdn@tweag-laptop:~/tweag/jupyterWith/depot]$ ~/tweag/julia2nix/julia2nix 
nix-pre-commit-hooks: updating /home/bakerdn/tweag/jupyterWith/depot repo
pre-commit uninstalled
pre-commit installed at .git/hooks/pre-commit
this derivation will be built:
  /nix/store/38vbsn9c64qzyvd7mf370hlk387cxxcp-python3-3.9.12-env.drv
these 4 paths will be fetched (0.34 MiB download, 1.48 MiB unpacked):
  /nix/store/0nvxrbjalaa2aqfncsriydbh52szdlax-python3.9-gitpython-3.1.24
  /nix/store/90ai6jq6wdd0g7hpk6s8w45q21cq9fyz-python3.9-gitdb-4.0.7
  /nix/store/h399rzl3fa9rszs340li1zvm1x6n8ym4-python3.9-smmap-4.0.0
  /nix/store/r3gxywc7svqlhb635h2kj2n51mzgr0yk-python3.9-ddt-1.4.4
copying path '/nix/store/h399rzl3fa9rszs340li1zvm1x6n8ym4-python3.9-smmap-4.0.0' from 'https://cache.nixos.org'...
copying path '/nix/store/r3gxywc7svqlhb635h2kj2n51mzgr0yk-python3.9-ddt-1.4.4' from 'https://cache.nixos.org'...
copying path '/nix/store/90ai6jq6wdd0g7hpk6s8w45q21cq9fyz-python3.9-gitdb-4.0.7' from 'https://cache.nixos.org'...
copying path '/nix/store/0nvxrbjalaa2aqfncsriydbh52szdlax-python3.9-gitpython-3.1.24' from 'https://cache.nixos.org'...
building '/nix/store/38vbsn9c64qzyvd7mf370hlk387cxxcp-python3-3.9.12-env.drv'...
created 237 symlinks in user environment
nix-pre-commit-hooks: hooks up to date
Cloning https://github.com/JuliaRegistries/General.git to /tmp/nix-shell.2ChYH5/nix-shell.37R8ZN/tmpvjx7gpf5/registry
Got registry revision ae45bf266152a01b6ec48462bbf14da79f405681
Got registry sha256 0lfjc2lwkg9w1lfg6jql5l8w4if3i5rn2hcsj7bafvslfyggd5g7
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/bakerdn/tweag/julia2nix/gather_packages.py", line 61, in process_item
    uuid = details[0]["uuid"]
TypeError: string indices must be integers
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/bakerdn/tweag/julia2nix/gather_packages.py", line 107, in <module>
    formatted_dicts = p.map(process_item, manifest.keys())
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
TypeError: string indices must be integers
Traceback (most recent call last):
  File "/home/bakerdn/tweag/julia2nix/julia2nix", line 45, in <module>
    main()
  File "/home/bakerdn/tweag/julia2nix/julia2nix", line 40, in main
    packages_nix = subprocess.check_output([script_dir.joinpath("gather_packages.py"), environment_dir, repo_url, str(workers)]).decode()
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '[PosixPath('/home/bakerdn/tweag/julia2nix/gather_packages.py'), PosixPath('.'), 'https://github.com/JuliaRegistries/General.git', '8']' returned non-zero exit status 1.

Expected behavior julia2nix to finish and create packages.nix, common.nix, default.nix, and fetchgit. It appears the only file not created is packages.nix.

Environment

GTrunSec commented 2 years ago

please try julia2nix.jl instead

thomasjm commented 2 years ago

This should be fixed now! We had to support the new manifest file format.

I have this working in CI with Julia 1.6 and 1.8. I assume 1.7 works too, but it's not in Nixpkgs so I don't have it tested under CI. Lmk if you have any problems!