Enabling Plan mode on pyvroom

[juanluisrto]

When calling problem.check() to use plan mode pyvroom complains that it has not been compiled with libglpk

I have added a flag to fix this, but when I compile pyvroom via conan and try import it in my notebook I get the following Importerror: symbol not found in flat namespace '__ZN5vroom10validation17check_and_set_ETAERKNS_5InputEj'

It seems to me that some bindings might be missing, but I am not sure. I think this is the line which is failing.

@jonathf do you see any quick fix for this?

[jonathf]

Thanks for giving it a shot.

All source files in use have to be included. With the use of GLPK some new files have to be added.

I've pushed a couple of missing includes to your branch that solves your error.

Looks like it complains that GLPK is missing now, so next step is to add installation of that to github action. Are you up for the task?

[codecov[bot]]

Codecov Report

Merging #84 (98b574b) into main (9880718) will increase coverage by 0.7%. Report is 6 commits behind head on main. The diff coverage is n/a.

@@           Coverage Diff           @@
##            main     #84     +/-   ##
=======================================
+ Coverage   78.2%   78.9%   +0.7%     
=======================================
  Files         29      29             
  Lines       1522    1545     +23     
  Branches      17      18      +1     
=======================================
+ Hits        1191    1220     +29     
+ Misses       322     316      -6     
  Partials       9       9             

Flag	Coverage Δ
binding	69.4% <ø> (+1.2%)	:arrow_up:
python	91.4% <ø> (+0.2%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

[juanluisrto]

@jonathf I managed to add glpk to the action and make the tests pass, however I am still having trouble to install it in my local machine (macos), therefore I don't know if its working properly. I can compile and install just fine, but when I import vroom I get this error

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../pyvroom/src/vroom/__init__.py", line 4, in <module>
    from ._vroom import _main, JOB_TYPE, STEP_TYPE  # type: ignore
ImportError: dlopen(.../pyvroom/src/vroom/_vroom.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_glp_add_cols'

I am still exploring the issue, the problem is probably in the linking with the glpk library on my machine. I installed it via brew. I am playing with the setup.py file a bit. Have you encountered this kind of issues before?

[jonathf]

I'm familiar with linker errors, and that might what you got, but in isolation your error doesn't tell me much. Doesn't look like your code changes affect the macos pipeline.

Anyways, I've updated the code now so it build on Linux and Macos in CI. You can try it out with pip install pyvroom==1.13.3-dev2. Works on my linux machine, can you confirm the same on your macos?

The only thing I couldn`t get working is Conan on Windows. @nilsnolde, do you think it is feasable to get this part working?

[juanluisrto]

Anyways, I've updated the code now so it build on Linux and Macos in CI. You can try it out with pip install pyvroom==1.13.3-dev2. Works on my linux machine, can you confirm the same on your macos?

I cannot install it via pip since the release step was omitted because compilation windows in didn't succeed. So the 1.13.3-dev2 version cannot be found in pypy.

I keep trying to compile it myself I am still failing to link glpk properly 😞

[jonathf]

Ah, right. My wheel was built locally. I've pushed version 1.13.3-dev4 now with windows disabled. It should therefore be installable soon.

[nilsnolde]

Ok seems like I got win to build. I can't push to this branch though @juanluisrto , so I'll PR to your fork as soon as the build completes entirely.

The workflow file only needed to be updated to build glpk with conan as well, as it doesn't have binaries on their platform. Before it was only building openssl.

[nilsnolde]

It takes forever though.. to build glpk conan first has to be build libtool. Good that we cache:)

[juanluisrto]

Thanks @nilsnolde for the windows fix.

Still, I am not able to conan compile and install pyvroom in my machine (macos M1) from 98b574b. Now I cannot even import the library since the glpk function glp_add_cols is not found. Maybe we might not be using the right glpk version?

[nilsnolde]

Are you installing with conan? Locally I have the same/similar version installed (via my package manager on arch linux): glpk 5.0-2. For me, the tests pass. Maybe you can try the brew version?

[nilsnolde]

If you can come up with a standalone, minimal example so we can try, that'd help. I'm inclined to think it's M1 specific, but maybe I'm too prejudiced by now :D

[juanluisrto]

I did not manage to use glpk from the brew installation so I am sticking to conan for now.

This is a minimal example of what I am doing

conan install --build=missing --install-folder conan_build .
pip install .
python -c "import vroom"

This is what I get in my shell. The dependencies installation via conan and pyvroom installation via pip don't raise errors, but there is something missing. Not sure if pip is aware of / using properly conan dependencies.

pyvroom git:(bugfix/plan-mode) ✗ conan install --build=missing --install-folder conan_build . && pip install . && python -c "import vroom"
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=14
os=Macos
os_build=Macos
[options]
[build_requires]
[env]

conanfile.txt: Installing package
Requirements
    asio/1.21.0 from 'conancenter' - Cache
    glpk/5.0 from 'conancenter' - Cache
    openssl/1.1.1m from 'conancenter' - Cache
Packages
    asio/1.21.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    glpk/5.0:6841fe8f0f22f6fa260da36a43a94ab525c7ed8d - Cache
    openssl/1.1.1m:6841fe8f0f22f6fa260da36a43a94ab525c7ed8d - Cache

Installing (downloading, building) binaries...
asio/1.21.0: Already installed!
glpk/5.0: Already installed!
openssl/1.1.1m: Already installed!
conanfile.txt: Generator txt created conanbuildinfo.txt
conanfile.txt: Generator json created conanbuildinfo.json
conanfile.txt: Aggregating env generators
conanfile.txt: Generated conaninfo.txt
conanfile.txt: Generated graphinfo

  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy in miniconda3/envs/myenv/lib/python3.10/site-packages (from pyvroom==1.13.3.dev16+g98b574b) (1.24.3)
Requirement already satisfied: pandas in miniconda3/envs/myenv/lib/python3.10/site-packages (from pyvroom==1.13.3.dev16+g98b574b) (2.0.2)
Requirement already satisfied: python-dateutil>=2.8.2 in miniconda3/envs/myenv/lib/python3.10/site-packages (from pandas->pyvroom==1.13.3.dev16+g98b574b) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in miniconda3/envs/myenv/lib/python3.10/site-packages (from pandas->pyvroom==1.13.3.dev16+g98b574b) (2023.3)
Requirement already satisfied: tzdata>=2022.1 in miniconda3/envs/myenv/lib/python3.10/site-packages (from pandas->pyvroom==1.13.3.dev16+g98b574b) (2023.3)
Requirement already satisfied: six>=1.5 in miniconda3/envs/myenv/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->pyvroom==1.13.3.dev16+g98b574b) (1.16.0)
Building wheels for collected packages: pyvroom
  Building wheel for pyvroom (pyproject.toml) ... done
  Created wheel for pyvroom: filename=pyvroom-1.13.3.dev16+g98b574b-cp310-cp310-macosx_11_0_arm64.whl size=826952 sha256=6e3bcab01f52eaf68d0a836330585423913403610421ff59d747f4f7b70bd21d
  Stored in directory: /private/var/folders/y5/c_b8k98d3kj7067r3kl2t9j80000gp/T/pip-ephem-wheel-cache-5ofsjh_i/wheels/64/1c/40/055bc9f014cb116cace38a5924a0a9ec4154aaa24b83fa7c00
Successfully built pyvroom
Installing collected packages: pyvroom
Successfully installed pyvroom-1.13.3.dev16+g98b574b

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "miniconda3/envs/myenv/lib/python3.10/site-packages/vroom/__init__.py", line 4, in <module>
    from ._vroom import _main, JOB_TYPE, STEP_TYPE  # type: ignore
ImportError: dlopen(miniconda3/envs/myenv/lib/python3.10/site-packages/vroom/_vroom.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_glp_add_cols'

[nilsnolde]

Yeah sorry, you did give a minimal example before. And I think you're right, might be that OSX needs to be explicitly told where to search for libraries when linking. Maybe this helps:

diff --git a/setup.py b/setup.py
index 3f429d4..1151cf4 100644
--- a/setup.py
+++ b/setup.py
@@ -56,7 +56,7 @@ else:  # anything *nix
 # try conan dependency resolution
 conanfile = tuple(Path(__file__).parent.resolve().rglob("conanbuildinfo.json"))
 if conanfile:
-    logging.info("Using conan to resolve dependencies.")
+    logging.critical("Using conan to resolve dependencies.")
     with conanfile[0].open() as f:
         conan_deps = json.load(f)["dependencies"]
     for dep in conan_deps:
@@ -64,6 +64,8 @@ if conanfile:
         libraries.extend(dep["libs"])
         libraries.extend(dep["system_libs"])
         library_dirs.extend(dep["lib_paths"])
+        for lib_path in dep["lib_paths"]:
+            extra_link_args.insert(0, f"-L{lib_path}")
 else:
     logging.warning("Conan not installed and/or no conan build detected. Assuming dependencies are installed.")

And then build with pip install . -vvv and look for "Using conan to resolve dependencies." in stdout. Just to make sure conan is being used.