Add installation instructions to README.md

murrayrm commented 1 week ago

I was able to install flowsynth and run the unit tests, but it would be nice to have installation instructions in the README file.

Here's what I did:

conda create -n python3.10-tulip python=3.10
pip install -e .
Manual installation of spot from https://spot.lre.epita.fr
cd src; pytest ../tests

A couple of notes:

It looks like flowsynth requires Python 3.10 (or perhaps less) since Gurobi is not supported for Python 3.11 and higher (according to pip).
The spot package had to be installed manually (rather than via pip or conda)

jgraeb commented 1 week ago

I added installation instructions to the documentation and a link in the readme in #5 . Please let me know if these new instructions work for you.

slivingston commented 1 week ago

Official Gurobi documentation indicates that version 10 supports Python 3.10 and 3.11, while version 11 supports up to Python 3.12 (the most recent Python version): https://support.gurobi.com/hc/en-us/articles/360013195212-Which-Python-versions-are-supported-by-Gurobi

Floras (or, the package formerly known as "flowsynth") is pinned at gurobipy 10: https://github.com/tulip-control/floras/blob/37cf9d007a38d8dd2eba8d860daf5ac873f6e188/pyproject.toml#L20

@jgraeb is it possible to upgrade to gurobipy version 11 ?

jgraeb commented 1 week ago

I am trying to make it work. Gurobi 11 installs fine, but I get this error message when installing spot (when testing using GitHub actions main.yaml):

buddy_wrap.cxx:199:11: fatal error: Python.h: No such file or directory
  199 | # include <Python.h>
      |           ^~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:1435: buddy_wrap.lo] Error 1
make[1]: Leaving directory '/home/runner/work/floras/floras/spot-2.12/python'
make: *** [Makefile:1350: install-recursive] Error 1

Any idea what this could be? python3-dev is the newest version. The only change is the pyproject.toml file and lock file changing the python version to >=3.10 and Gurobi to 11.0.3.

slivingston commented 1 week ago

On Debian hosts, you need the libpython*-dev package corresponding to the Python version. Here, if you are building with Python 3.10, try

apt install libpython3.10-dev

(libpython3.11-dev if you are building with Python 3.11 etc.)

jgraeb commented 1 week ago

Ok I added this but get the same error message here: https://github.com/jgraeb/floras/actions/runs/11154250994/job/31003212287 Also what is the reason that wasn't needed before the change?

slivingston commented 1 week ago

My mistake: I assumed you were referring to a local installation. In CI jobs, this should already be installed and therefore is likely not the reason for the error. I will think more and follow up soon.

slivingston commented 1 week ago

I reverted the addition of libpython*-dev and removed matplotlib from pyproject.yaml (it was imported but not used anywhere). Now CI jobs are passing with gurobipy version 11 on my branch: https://github.com/slivingston/floras/tree/j-main I can open a PR with these, or you can apply them on your own branch.

jgraeb commented 1 week ago

Ok, thanks. Matplotlib wasn't used because I haven't added the plotting functions yet, so I will need to add it back soon. Do you know what the reason for the spot error message was involving matplotlib?

slivingston commented 1 week ago

There was some version compatibility problem between matplotlib and numpy because PDM switched to numpy version 2 in the conda CI build. I just added matplotlib again, now with a version bound of matplotlib>=3.9.2, and it passes: https://github.com/slivingston/floras/actions/runs/11156083249

slivingston commented 1 week ago

...and for pdm install, PDM automatically updates dependencies within the specified version ranges. There is no upper bound on the numpy dependency, so it jumps to version 2 (e.g., as in my recent CI job). Confusingly, if we do not want this behavior, we should install dependencies using pdm sync instead: https://pdm-project.org/latest/usage/lockfile/#install-the-packages-pinned-in-lock-file

jgraeb commented 1 week ago

That error with the numpy in conda environment is fixed by updating the versions you stated. But even after updating the versions and trying to step through your changes (versions and use apt-get), I still have the same problem when installing spot in the pdm environment in main.yaml https://github.com/jgraeb/floras/actions/runs/11156329821/job/31008733300.

slivingston commented 1 week ago

@jgraeb something seems to be broken with the CI cache for Spot. Can you try to delete the cache item with "spot" in its name at https://github.com/jgraebfloras/actions/caches and then re-run https://github.com/jgraeb/floras/actions/runs/11156329821 ?

slivingston commented 1 week ago

The commit where the CI jobs started to fail changed the Python versions expression in pyproject.toml. My hypothesis is this resulted in a new venv generated by PDM, but the cache key for Spot is not a function of this, and therefore, make install was called for the cached Spot that has the old paths, hence the error of not finding Python.h

If the hypothesis is true, then the solution is to add another hashFiles() to the cache key in main.yaml:

${{ runner.os }}-${{ matrix.python-version }}-spot-${{ hashFiles('get_spot.py') }}-${{ hashFiles('pyproject.toml') }}

or hashFiles('pdm.lock')

jgraeb commented 1 week ago

That fixed it in #6 .

murrayrm commented 6 days ago

I tried reinstalling following the directions here. I got an error when pdm tried to install pygraphviz:

  ✖ Install pygraphviz 1.14 failed
  ✖ Install pygraphviz 1.14 failed

ERRORS:
add pygraphviz failed:
Traceback (most recent call last):
  File 
"/Users/murray/miniconda3/envs/python3.12-tulip/lib/python3.12/concurrent/future
s/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/Users/murray/miniconda3/envs/python3.12-tulip/lib/python3.12/site-packages/pdm
/installers/synchronizers.py", line 29, in install_candidate
    self.manager.install(can)
  File 
"/Users/murray/miniconda3/envs/python3.12-tulip/lib/python3.12/site-packages/pdm
/installers/manager.py", line 33, in install
    prepared.build(),
    ^^^^^^^^^^^^^^^^
  File 
"/Users/murray/miniconda3/envs/python3.12-tulip/lib/python3.12/site-packages/pdm
/models/candidates.py", line 411, in build
    self._cached = Path(builder.build(build_dir, 
metadata_directory=self._metadata_dir))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^
  File 
"/Users/murray/miniconda3/envs/python3.12-tulip/lib/python3.12/site-packages/pdm
/builders/base.py", line 84, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File 
"/Users/murray/miniconda3/envs/python3.12-tulip/lib/python3.12/site-packages/pdm
/builders/wheel.py", line 26, in build
    filename = self._hook.build_wheel(out_dir, self.config_settings, 
metadata_directory)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^
  File 
"/Users/murray/miniconda3/envs/python3.12-tulip/lib/python3.12/site-packages/pyp
roject_hooks/_impl.py", line 256, in build_wheel
    return self._call_hook(
           ^^^^^^^^^^^^^^^^
  File 
"/Users/murray/miniconda3/envs/python3.12-tulip/lib/python3.12/site-packages/pyp
roject_hooks/_impl.py", line 392, in _call_hook
    self._subprocess_runner(
  File 
"/Users/murray/miniconda3/envs/python3.12-tulip/lib/python3.12/site-packages/pdm
/builders/base.py", line 281, in subprocess_runner
    return log_subprocessor(cmd, cwd, extra_environ=env)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/Users/murray/miniconda3/envs/python3.12-tulip/lib/python3.12/site-packages/pdm
/builders/base.py", line 128, in log_subprocessor
    raise build_error(e) from None
pdm.exceptions.BuildError: Build backend raised error: Showing the last 10 lines
of the build output:
#define SWIG_PYTHON_STRICT_BYTE_CHAR
        ^
<command line>:2:9: note: previous definition is here
#define SWIG_PYTHON_STRICT_BYTE_CHAR 1
        ^
pygraphviz/graphviz_wrap.c:3023:10: fatal error: 'graphviz/cgraph.h' file not 
found
#include "graphviz/cgraph.h"
         ^~~~~~~~~~~~~~~~~~~
1 warning and 1 error generated.
error: command '/usr/bin/clang' failed with exit code 1

  0:00:00 ✖ Some package operations failed. 1/1
See /Users/murray/Library/Logs/pdm/pdm-install-uamoq43k.log for detailed debug 
log.
[InstallationError]: Some package operations failed.

It looks like the error is that graphviz is not found, so I installed that manually (using conda). That fixed the problem and installation of floras completed successfully.

I next tried to install spot using conda (since this is faster). That failed on my machine:

(python3.12-tulip) murray@tekisic floras % conda install -c conda-forge spot
Channels:
 - conda-forge
 - defaults
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - spot

Current channels:

  - https://conda.anaconda.org/conda-forge
  - defaults

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

I then installed spot using pdm. This took a long time, so I wish that conda worked...

At this stage, I was able to run pytest tests.

Things to address based on my experience:

Update graphviz dependency or include instructions to pre-install (this may be MacOS specific, since it seems to work in the CI tests??).
Update spot documentation to indicate it is not available on all platforms (e.g. osx-arm64, apparently).

Some other comments:

I would encourage putting the installation instructions in the README file, even if they are redundant. It is a bit of a pain having to find them on readthedocs.
I would encourage putting a copy of the installation instructions on the first page of the documentation (instead of having to click through to a separate page).
I find it odd to have the installation instructions on a page titled "Contributing to Floras".

jgraeb commented 5 days ago

I updated the instruction on readthedocs and added them to the readme. I did not include them in the readthedocs welcome page yet as they are extremely long - hopefully they will become more streamlined (`pip install floras') soon and then I will add them to the welcome page. I am investigating the installation on macOS with pygraphviz. The test using conda runs successfully, but using pdm and preinstalling pygraphviz using pip results in an error.

slivingston commented 5 days ago

I can try to debug it, too, but have you considered using the Python package graphviz instead? It is more popular (in terms of stars on GitHub), and has been used successfully in TuLiP and dd.

jgraeb commented 3 days ago

Originally I had graphviz and pygraphviz but was able to remove graphviz, which worked for all environments except macOS and pdm (but that one does not work yet anyways). I just tried using just graphviz instead, but I need pygraphviz to plot the graphs using networkx and the interface they provide, so unfortunately it needs to be pygraphviz.

The problem is: When using macOS (without conda) installing pygraphviz does not work through pdm - that's why I tried to install it using brew and pip and it gets installed successfully (there seem to be some difficulties with pygraphviz and macOS). But then it does not find the pygraphviz installation when it installs floras using pdm and errors out. When pygraphviz is installed using conda, pdm then finds the installation and successfully installs floras. We could require macOS users to preinstall pygraphviz, but I do not want to change the pyptoject.toml file to remove pygraphviz for a different os. You can see the details of the run here, where it installs graphviz but then does not find it.

slivingston commented 10 hours ago

I proposed a change to the CI workflow file that works for PDM on macOS (without conda) at https://github.com/tulip-control/floras/pull/7/files#r1796527847

tulip-control / floras

Add installation instructions to README.md #2