Update Dockerfile - Githubissues

a private ssh key is no longer necessary, since everything is on public github.com.

however, I do get the following error, having uncommented the final line:

STEP 11/11: RUN python3 -c "import qiskit_alt; qiskit_alt.compile_qiskit_alt()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/qiskit_alt/qiskit_alt/__init__.py", line 1, in <module>
    from ._julia_project import julia, compile_qiskit_alt, update_qiskit_alt
  File "/qiskit_alt/qiskit_alt/_julia_project.py", line 4, in <module>
    from julia_project import JuliaProject
  File "/usr/local/lib/python3.9/site-packages/julia_project/__init__.py", line 9, in <module>
    os.environ['PYTHON'] = shutil.which("python")
  File "/usr/lib64/python3.9/os.py", line 684, in __setitem__
    value = self.encodevalue(value)
  File "/usr/lib64/python3.9/os.py", line 756, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType
Error: error building at STEP "RUN python3 -c "import qiskit_alt; qiskit_alt.compile_qiskit_alt()"": error while running runtime: exit status 1

If you are using a recent version of the main branch that explains the errors. The API has changed. The user has to explicitly init qiskit_alt. tkf convinced me this is the way to go. I changed the README to reflect this in most places. Now you do

import qiskit_alt; qiskit_alt.project.ensure_init(); qiskit_alt.project.compile()

Thanks. I noticed the updated README shortly after I posted.

I have pushed my latest Dockerfile and am now getting a new error about import PyCall failing. I think the Pkg.instantiate() needs to happen somewhere in ensure_init().

STEP 10/11: RUN printf "y\nn\n" | python3 -c "import qiskit_alt; qiskit_alt.project.ensure_init()"

I can compile a system image after installation.
Compilation may take a few, or many, minutues. You may compile now, later, or never.
Would you like to compile a system image after installation?
 [Y/n] 
You can install all of the Julia packages and package information in a module-specific "depot",
that is, one specific to this Python module. This may allow you to use Julia with python projects
that have different Python installation locations.
Or you can install packages in the standard per-user Julia "depot".
Would you like to use a python-module-specific depot for Julia packages?
 [Y/n] [ Info: Julia version info
Julia Version 1.6.5
Commit 9058264a69 (2021-12-19 12:30 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  uname: Linux 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06 EST 2022 x86_64 x86_64
  CPU: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz: 
                 speed         user         nice          sys         idle          irq
       #1-12  2600 MHz    5368369 s     483810 s    1074572 s    6256370 s     228300 s

  Memory: 31.062366485595703 GB (2790.8671875 MB free)
  Uptime: 1.86061161e6 sec
  Load Avg:  2.44  1.49  1.53
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  PYTHON_JULIACALL_NOINIT = yes
  JULIA_PYTHONCALL_EXE = /usr/bin/python3
  JULIA_PYTHONCALL_LIBPTR = 140666852106720
  JULIA_PROJECT = /root/.julia/julia_project/myjuliamod-1.6.5
  HOME = /root
  PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[ Info: Julia executable: /usr/bin/julia
[ Info: Trying to import PyCall...
┌ Error: `import PyCall` failed
│   exception =
│    ArgumentError: Package PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] is required but does not seem to be installed:
│     - Run `Pkg.instantiate()` to install all recorded dependencies.
│    
│    Stacktrace:
│     [1] top-level scope
│       @ /usr/local/lib/python3.9/site-packages/julia/install.jl:36
└ @ Main /usr/local/lib/python3.9/site-packages/julia/install.jl:38
[ Info: Installing PyCall...
    Updating registry at `~/.julia/registries/QuantumRegistry`
    Updating git-repo `https://github.com/Qiskit-Extensions/QuantumRegistry.git`
   Resolving package versions...
ERROR: LoadError: expected package `PyCall [438e738f]` to be registered
Stacktrace:
 [1] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Iterators.Pairs{Symbol, Base.PipeEndpoint, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.PipeEndpoint}}})
   @ Pkg.API /builddir/build/BUILD/julia-1.6.5/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:204
 [2] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.PipeEndpoint, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /builddir/build/BUILD/julia-1.6.5/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:80
 [3] top-level scope
   @ /usr/local/lib/python3.9/site-packages/julia/install.jl:118
in expression starting at /usr/local/lib/python3.9/site-packages/julia/install.jl:73
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/julia_project/_julia_project.py", line 189, in ensure_init
    self.init()
  File "/usr/local/lib/python3.9/site-packages/julia_project/_julia_project.py", line 253, in init
    self.calljulia.start_julia()
  File "/usr/local/lib/python3.9/site-packages/julia_project/pyjulia.py", line 154, in start_julia
    self.init_julia_module()
  File "/usr/local/lib/python3.9/site-packages/julia_project/pyjulia.py", line 120, in init_julia_module
    julia.install(julia=self.julia_path)
  File "/usr/local/lib/python3.9/site-packages/julia/tools.py", line 118, in install
    raise PyCallInstallError("Installing", output)
julia.tools.PyCallInstallError: Installing PyCall failed.

** Important information from Julia may be printed before Python's Traceback **

Some useful information may also be stored in the build log file
`~/.julia/packages/PyCall/*/deps/build.log`.

Error: error building at STEP "RUN printf "y\nn\n" | python3 -c "import qiskit_alt; qiskit_alt.project.ensure_init()"": error while running runtime: exit status 1

I introduced this bug while trying to catch more libpython mismatch errors. The error you get above

LoadError: expected package `PyCall [438e738f]` to be registered

is due to the registry not being installed in the new, empty, depot before resolve and instantiate are called. I introduced this because the python module julia does not use a project to load PyCall, it uses the lower level require. Furthermore, you are stuck with whatever project was active when import julia happened ^*. Later, when I compile the system image, our desired project is active. This can (and did) cause version skew and libpython skew and a segfault. So, in between the time that libjulia is initialized and PyCall is loaded, I activate a new project, so that the desired PyCall is found. But, I neglected to make sure the registries are installed first.

I have been trying to use the default depot (~/.julia) by default and detect libpython incompatibility and take some action. I am strongly considering just making a "private" depot the default. This could make the logic much less complicated. The code is becoming hairier and more complicated and the game of whack-a-mole is (probably) not over.

In fact, last night, I wrote quite a bit of code to do the installation by calling from python new julia processes with command line args. There is no reason that it all has to happen in one process, with all the calls to jl_blah_blah and byte and utf8 conversions, etc. etc. This is a new thing, and more work. But, I think in the end it will be less work (eg. in maintenance) than continuing the current approach.

* It just occurred to me that I can set up the project before doing import julia. This could be done by setting an environment variable.

This should be fixed with the recent versions (last day or so) of qiskit_alt / julia_project. Kristoffer Carlsson suggested that is actually a bug in Pkg.resolve

I just merged in the latest main, but I am still getting an error.

$ docker build --rm=false .
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
STEP 1/11: FROM fedora:34
STEP 2/11: RUN dnf install -y git @development-tools gcc-c++ python3 python3-devel julia
--> Using cache 83c7976578f876c8ff68066bee286b9532ecd44a99aeca8aeaedb712400df826
--> 83c7976578f
STEP 3/11: RUN python3 -m pip install -U pip
--> Using cache 3962627037835e36bd953bc0fe23f2d7416a31b01cce7195bf43f8390f5220f5
--> 39626270378
STEP 4/11: WORKDIR /qiskit_alt
--> Using cache 29e3c699fb026b8be345f56eac85199999ec62d470d22448b6fb4ba3135e8cf9
--> 29e3c699fb0
STEP 5/11: COPY requirements.txt .
--> Using cache e242edd0ac9ce4858922a830fb693c8e90f22fda145b402ff8e133b1eae8b740
--> e242edd0ac9
STEP 6/11: RUN pip install -r requirements.txt
--> Using cache 5dc18a29ca92b16b0c56161defd0cbc7c662fdd6928767881bf2eec8752725fc
--> 5dc18a29ca9
STEP 7/11: RUN julia -e 'using Pkg; pkg"registry add https://github.com/Qiskit-Extensions/QuantumRegistry.git"'
--> Using cache fa8f8e7ca4469baf7e8283a8d006201c5a1e06e1fbc77b2cd4069a37723152aa
--> fa8f8e7ca44
STEP 8/11: COPY . .
--> 452cf551095
STEP 9/11: RUN pip install -e .
Obtaining file:///qiskit_alt
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: juliacall in /usr/local/lib/python3.9/site-packages (from qiskit-alt==0.1.3) (0.6.1)
Collecting julia-project>=0.1.12
  Downloading julia_project-0.1.13-py3-none-any.whl (30 kB)
Requirement already satisfied: pyscf~=2.0 in /usr/local/lib64/python3.9/site-packages (from qiskit-alt==0.1.3) (2.0.1)
Requirement already satisfied: qiskit-terra~=0.19 in /usr/local/lib64/python3.9/site-packages (from qiskit-alt==0.1.3) (0.19.2)
Requirement already satisfied: julia in /usr/local/lib/python3.9/site-packages (from qiskit-alt==0.1.3) (0.5.7)
Requirement already satisfied: qiskit-nature~=0.2 in /usr/local/lib/python3.9/site-packages (from qiskit-alt==0.1.3) (0.3.1)
Collecting tomli
  Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting find-libpython
  Downloading find_libpython-0.2.0-py3-none-any.whl (7.4 kB)
Collecting find-julia>=0.2.2
  Downloading find_julia-0.2.2-py3-none-any.whl (11 kB)
Requirement already satisfied: juliapkg~=0.1.0 in /usr/local/lib/python3.9/site-packages (from juliacall->qiskit-alt==0.1.3) (0.1.2)
Requirement already satisfied: scipy!=1.5.0,!=1.5.1 in /usr/local/lib64/python3.9/site-packages (from pyscf~=2.0->qiskit-alt==0.1.3) (1.8.0)
Requirement already satisfied: numpy!=1.16,!=1.17,>=1.13 in /usr/local/lib64/python3.9/site-packages (from pyscf~=2.0->qiskit-alt==0.1.3) (1.22.3)
Requirement already satisfied: h5py>=2.7 in /usr/local/lib64/python3.9/site-packages (from pyscf~=2.0->qiskit-alt==0.1.3) (3.6.0)
Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib64/python3.9/site-packages (from qiskit-nature~=0.2->qiskit-alt==0.1.3) (1.0.2)
Requirement already satisfied: setuptools>=40.1.0 in /usr/lib/python3.9/site-packages (from qiskit-nature~=0.2->qiskit-alt==0.1.3) (53.0.0)
Requirement already satisfied: retworkx>=0.10.1 in /usr/local/lib64/python3.9/site-packages (from qiskit-nature~=0.2->qiskit-alt==0.1.3) (0.11.0)
Requirement already satisfied: psutil>=5 in /usr/local/lib64/python3.9/site-packages (from qiskit-nature~=0.2->qiskit-alt==0.1.3) (5.9.0)
Requirement already satisfied: python-constraint>=1.4 in /usr/local/lib/python3.9/site-packages (from qiskit-terra~=0.19->qiskit-alt==0.1.3) (1.4.0)
Requirement already satisfied: stevedore>=3.0.0 in /usr/local/lib/python3.9/site-packages (from qiskit-terra~=0.19->qiskit-alt==0.1.3) (3.5.0)
Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.9/site-packages (from qiskit-terra~=0.19->qiskit-alt==0.1.3) (2.8.2)
Requirement already satisfied: ply>=3.10 in /usr/local/lib/python3.9/site-packages (from qiskit-terra~=0.19->qiskit-alt==0.1.3) (3.11)
Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.9/site-packages (from qiskit-terra~=0.19->qiskit-alt==0.1.3) (0.3.4)
Requirement already satisfied: symengine>=0.8 in /usr/local/lib64/python3.9/site-packages (from qiskit-terra~=0.19->qiskit-alt==0.1.3) (0.9.2)
Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.9/site-packages (from qiskit-terra~=0.19->qiskit-alt==0.1.3) (1.10)
Requirement already satisfied: tweedledum<2.0,>=1.1 in /usr/local/lib64/python3.9/site-packages (from qiskit-terra~=0.19->qiskit-alt==0.1.3) (1.1.1)
Requirement already satisfied: jill in /usr/local/lib/python3.9/site-packages (from find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (0.11.0)
Requirement already satisfied: julia-semver in /usr/local/lib/python3.9/site-packages (from find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (0.1.3)
Requirement already satisfied: semantic-version~=2.9 in /usr/local/lib/python3.9/site-packages (from juliapkg~=0.1.0->juliacall->qiskit-alt==0.1.3) (2.9.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/site-packages (from python-dateutil>=2.8.0->qiskit-terra~=0.19->qiskit-alt==0.1.3) (1.16.0)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.9/site-packages (from scikit-learn>=0.20.0->qiskit-nature~=0.2->qiskit-alt==0.1.3) (1.1.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.9/site-packages (from scikit-learn>=0.20.0->qiskit-nature~=0.2->qiskit-alt==0.1.3) (3.1.0)
Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/local/lib/python3.9/site-packages (from stevedore>=3.0.0->qiskit-terra~=0.19->qiskit-alt==0.1.3) (5.8.1)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.9/site-packages (from sympy>=1.3->qiskit-terra~=0.19->qiskit-alt==0.1.3) (1.2.1)
Requirement already satisfied: python-gnupg in /usr/local/lib/python3.9/site-packages (from jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (0.4.8)
Requirement already satisfied: requests-futures in /usr/local/lib/python3.9/site-packages (from jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (1.0.0)
Requirement already satisfied: jsonschema in /usr/local/lib/python3.9/site-packages (from jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (4.4.0)
Requirement already satisfied: fire in /usr/local/lib/python3.9/site-packages (from jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (0.4.0)
Requirement already satisfied: requests in /usr/local/lib/python3.9/site-packages (from jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (2.27.1)
Requirement already satisfied: wget in /usr/local/lib/python3.9/site-packages (from jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (3.2)
Requirement already satisfied: termcolor in /usr/local/lib/python3.9/site-packages (from fire->jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (1.1.0)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.9/site-packages (from jsonschema->jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (21.4.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib64/python3.9/site-packages (from jsonschema->jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (0.18.1)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests->jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (2021.10.8)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests->jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/site-packages (from requests->jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (2.0.12)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests->jill->find-julia>=0.2.2->julia-project>=0.1.12->qiskit-alt==0.1.3) (1.26.8)
Installing collected packages: find-libpython, tomli, find-julia, julia-project, qiskit-alt
  Attempting uninstall: find-julia
    Found existing installation: find-julia 0.2.1
    Uninstalling find-julia-0.2.1:
      Successfully uninstalled find-julia-0.2.1
  Attempting uninstall: julia-project
    Found existing installation: julia-project 0.1.7
    Uninstalling julia-project-0.1.7:
      Successfully uninstalled julia-project-0.1.7
  Running setup.py develop for qiskit-alt
Successfully installed find-julia-0.2.2 find-libpython-0.2.0 julia-project-0.1.13 qiskit-alt tomli-2.0.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
--> 6db6e49fa4f
STEP 10/11: RUN printf "y\nn\n" | python3 -c "import qiskit_alt; qiskit_alt.project.ensure_init()"

I can compile a system image after installation.
Compilation may take a few, or many, minutues. You may compile now, later, or never.
Would you like to compile a system image after installation?
 [Y/n] 
You can install all of the Julia packages and package information in a
module-specific "depot", that is, one specific to this Python module. This may
allow you to use Julia with python projects that have different Python
installation locations.  If you answer "no", packages will be installed in the
standard per-user Julia "depot".

Would you like to use a python-module-specific depot for Julia packages?
 [Y/n] Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/julia_project/_julia_project.py", line 195, in ensure_init
    self.init()
  File "/usr/local/lib/python3.9/site-packages/julia_project/_julia_project.py", line 284, in init
    install.ensure_project_ready_fix_pycall(
  File "/usr/local/lib/python3.9/site-packages/julia_project/install.py", line 511, in ensure_project_ready_fix_pycall
    ensure_project_ready(project_path, julia_exe, depot_path=depot_path,
  File "/usr/local/lib/python3.9/site-packages/julia_project/install.py", line 323, in ensure_project_ready
    ensure_general_registry(project_path, julia_exe=julia_exe, depot_path=depot_path, clog=clog)
  File "/usr/local/lib/python3.9/site-packages/julia_project/install.py", line 193, in ensure_general_registry
    raise Exception("Installation of General registry failed.")
Exception: Installation of General registry failed.
Installing general registry
  Activating environment at `~/.julia/julia_project/qiskit_alt-1.6.5/Project.toml`
       Added registry `General` to `~/.julia/registries/General`
Error: error building at STEP "RUN printf "y\nn\n" | python3 -c "import qiskit_alt; qiskit_alt.project.ensure_init()"": error while running runtime: exit status 1

pip install -r requirements.txt

Perhaps I should remove this requirements.txt. I only use setup.cfg now. I have not kept it up to date. But, this should not be the source of this bug.

julia -e 'using Pkg; pkg"registry add https://github.com/Qiskit-Extensions/QuantumRegistry.git"'

This should not cause any problems. But, it is also not necessary because it is done during ensure_init. Furthermore, if a "private" depot is selected, then ensure_init will install QuantumRegistry there. Whereas, the line above will store it in ~/.julia.

Running pip as the 'root' user can result in broken permissions

There are two things that may be uncovering a bug because I have not tested them, although they are supposed to be supported: 1) installing as root (maybe you are doing this above?) and 2) using the user or system python rather than a virtual environment. I'll try these now. I'm a little leary of root though. I'd have to be sure I can reset the environment afterward. Maybe a chroot environment...
I see you are piping answers to the questions from ensure_init. You can instead set the environment variables (!! This has been broken for some time. I fixed it and just pushed a commit. I had inadvertently used the wrong prefix for the env vars.) The environment vars are described here

I think I found the problem. Julia 1.6 installs the general registry in a different way than 1.7. I am checking that installation succeeds by checking only the new way. Relying on Julia errors would be better perhaps. But, I have not figured out how to trap them when using python subprocess (probably not difficlut). So, I make ad hoc checks for failure points and throw errors.

I'll fix this one and push to main.

FYI: The following do work.

I just tried installing with no virtual env. That is, installing to the user's python installation ~/.local/lib .... This succeeded.

I also tried staring julia as root and installing the general registry. It seemed to install as expected.

Hopefully this is fixed with b7260e1

I also tagged qiskit_alt v0.1.4

I'll commit this now. It is, at worst, less broken than the current Dockerfile, and might even work. I can try to test it locally, as well. But first, I should push recent changes.

works for me on main. :slightly_smiling_face:

qiskit-community / qiskit-alt

Update Dockerfile #5