(Python) Installation issue

[joaquimg]

Hi, I am trying to run SpineOpt, but I get

[ Info: Running /mnt/c/Users/joaquimgarcia/Desktop/BT/open-energy-modeling-benchmarks/SpineOpt/cases/1_electrolyzer_with_rolling_horizon.json
ERROR: LoadError: UndefVarError: `_spinedb_api_not_found_error` not defined
Stacktrace:
 [1] top-level scope
   @ ~/.julia/packages/SpineInterface/OVrbG/src/api/db.jl:602
 [2] eval
   @ ./boot.jl:385 [inlined]
 [3] _import_spinedb_api()
   @ SpineInterface ~/.julia/packages/SpineInterface/OVrbG/src/api/db.jl:596
 [4] _create_db_handler
   @ ~/.julia/packages/SpineInterface/OVrbG/src/api/db.jl:556 [inlined]
 [5] open_connection(db_url::String)
   @ SpineInterface ~/.julia/packages/SpineInterface/OVrbG/src/api/db.jl:527
 [6] main(args::Vector{String})
   @ Main /mnt/c/Users/joaquimgarcia/Desktop/BT/open-energy-modeling-benchmarks/SpineOpt/main.jl:107

   ...

   caused by: PyError (PyImport_ImportModule

The Python package spinedb_api could not be imported by pyimport. Usually this means
that you did not install spinedb_api in the Python version being used by PyCall.

PyCall is currently configured to use the Python version at:

/usr/bin/python3

and you should use whatever mechanism you usually use (apt-get, pip, conda,
etcetera) to install the Python package containing the spinedb_api module.

One alternative is to re-configure PyCall to use a different Python
version on your system: set ENV["PYTHON"] to the path/name of the python
executable you want to use, run Pkg.build("PyCall"), and re-launch Julia.

Another alternative is to configure PyCall to use a Julia-specific Python
distribution via the Conda.jl package (which installs a private Anaconda
Python distribution), which has the advantage that packages can be installed
and kept up-to-date via Julia.  As explained in the PyCall documentation,
set ENV["PYTHON"]="", run Pkg.build("PyCall"), and re-launch Julia. Then,
To install the spinedb_api module, you can use `pyimport_conda("spinedb_api", PKG)`,
where PKG is the Anaconda package that contains the module spinedb_api,
or alternatively you can use the Conda package directly (via
`using Conda` followed by `Conda.add` etcetera).

) <class 'ModuleNotFoundError'>
ModuleNotFoundError("No module named 'spinedb_api'")

If I try to add spinedb_api with Conda, I get:

julia> Conda.add("spinedb_api")
[ Info: Running `conda install -y spinedb_api` in root environment
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - spinedb_api

Current channels:

  - https://conda.anaconda.org/conda-forge

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

ERROR: failed process: ...

[joaquimg]

I did found a workaround:

• use WSL instead of windows
• install libpq-dev with apt install libpq-dev (requires for psycopg2, in turn required for spinedb_api)
• install spinedb_api with pip (instead of Conda.jl / PyCall.jl)

Is this expected?

The docs say nothing about is the in the Julia REPL install: https://spine-tools.github.io/SpineOpt.jl/latest/getting_started/installation/#Installation-through-the-Julia-REPL

The docs just say I should have python in the manual install: https://spine-tools.github.io/SpineOpt.jl/latest/getting_started/installation/#7.-Configure-PyCall-through-the-Julia-REPL

[tarskul]

Thanks for working with spine tools!

Though, your issue is a bit unclear and confusing to me. You are talking about the installation process in the documentation but I got the feeling that you are taking different steps than in the instructions. From what I understand there may be a few things going on. In any case you should not need WSL (even though it is a cool program).

There may indeed be an issue with the manual installation of SpineOpt in the julia REPL, i.e. option 2 of the installation instructions. It is the only set of instructions that does not take into account the dependency of Spine Toolbox and the PyCall configuration. We'll have to take a closer look to make those instructions better. We also may need to add separate instructions for those who would like to install spineopt without spine toolbox (I think it may be sufficient to only install spine db api instead of spinetoolbox and for the rest follow the instructions and configuration of option 3 of the installation instructions).

Also for Conda we may need separate instructions. The instructions are mostly oriented towards the use of python and julia environments such that we do not need to ask of users to install conda. There is also no particular benefit for using conda (unless you are using more than just spine tools in that environment) as you need a separate environment for python and julia anyway (I vaguely recall that PyCall is not able to handle the situations of a single conda environment for python and julia). Regardless, conda can only be used as a replacement of the python environment, not the julia environment.

[datejada]

@joaquimg, just to confirm, to work with SpineInterface and SpineOpt you need to build PyCall. Sorry about that, I forgot about that part on my branch at the open-energy-modeling-benchmarks repo. Thanks for the feedback. I hope with your workaround is all working without a problem now.

I like @tarskul proposal to improve the docs. Thank you, too.

[joaquimg]

Hi @tarskul ,

Thanks for working with spine tools!

:)

You are talking about the installation process in the documentation but I got the feeling that you are taking different steps than in the instructions.

I tried both:

• https://spine-tools.github.io/SpineOpt.jl/latest/getting_started/installation/#Installation-through-the-Julia-REPL
• https://spine-tools.github.io/SpineOpt.jl/latest/getting_started/installation/#Installation-from-source

Then I tried fixing problems the command line was showing me.

In any case you should not need WSL (even though it is a cool program).

I just meant. I still haven't got it up and running in Windows. So I am using Linux (inside Windows).

There may indeed be an issue with the manual installation of SpineOpt in the julia REPL, i.e. option 2 of the installation instructions. It is the only set of instructions that does not take into account the dependency of Spine Toolbox and the PyCall configuration. We'll have to take a closer look to make those instructions better. We also may need to add separate instructions for those who would like to install spineopt without spine toolbox (I think it may be sufficient to only install spine db api instead of spinetoolbox and for the rest follow the instructions and configuration of option 3 of the installation instructions).

I imagined that following the manual installation of SpineOpt in the julia REPL, i.e. option 2 of the installation instructions would enable me to run simple scripts. But once I finish those step I cannot run scripts since I get the error of the original post.

There are steps missing in that options that include:

• python installed
• spinedb_api installed

Also for Conda we may need separate instructions. The instructions are mostly oriented towards the use of python and julia environments such that we do not need to ask of users to install conda. There is also no particular benefit for using conda (unless you are using more than just spine tools in that environment) as you need a separate environment for python and julia anyway (I vaguely recall that PyCall is not able to handle the situations of a single conda environment for python and julia). Regardless, conda can only be used as a replacement of the python environment, not the julia environment.

I just tried Conda because of the error message that spineopt threw (see original post):

...
or alternatively you can use the Conda package directly (via `using Conda` followed by `Conda.add` etcetera).

Which is probably just some rethrow from PyCall. Note that Conda here refers to the Conda julia wrapper used by PyCall.

Using Conda was not something I wanted, I was just following the recommendation of the error message.

I thinks the action item in this point is to use a julia try catch block to throw a proper message with the SpineOpt suggestion instead of the default PyCall message.

[joaquimg]

I just found the script: https://github.com/spine-tools/SpineOpt.jl/blob/master/.install_spinedb_api.jl in the root dir of this repo. Running this solved my issue in windows.

However, this still looks suboptimal as it seems to install the correct packages in my main Python environment.

Some steps for using the correct env in PyCall and installing these things in the correct env would be nice.

[tarskul]

You sure seem to like to stray from the standard routes as much as possible, don't you? XD That's ok ;)

However, I would not recommend using that script. Not only does it not respect environments, it also uses the custom spine registry which we have replaced by the official julia registry.

A full install of spine tools is about 2.2 GB which is not too bad on modern systems. So, you could install it the recommended way. But I also can understand that you may want a minimal setup. You should be able to follow option 3 of the installation instructions and pip install spine_db_api instead of spine toolbox.

If you like scripts, option 3 of the installation instructions is also fully implemented in the scripts in the spine installation tools repository. You would only have to make some minor tweaks to replace spine toolbox by the spine_db_api (and/or the dev version of spineopt by the julia registry version). Note that these scripts written for linux. On windows systems, it behaves like the 'naked man' from 'how I met your mother'; it works on 2/3 computers. ;) (which is why we don't mention the repository in the installation instructions.) At least it gives an overview of the commands you can do manually.