search

conda-forge / ceres-solver-feedstock

A conda-smithy repository for ceres-solver.
BSD 3-Clause "New" or "Revised" License
2 stars 10 forks source link

Windows: ceres-solver depends on suitesparse dlls but suitesparse comes with static .bins #3

Closed cboulay closed 5 years ago

cboulay commented 5 years ago

Issue:

It's great to finally have ceres-solver in conda!

I did conda install ceres-solver.

Using dependencies.exe, I see that the installed ceres.dll depends on many dlls, 4 of which are not included when installing ceres-solver: spqr.dll, cholmod.dll, openblas.dll, and cxsparse.dll.

There are static .bin files available for each of these (from suitesparse, I believe), but that's no help to ceres.dll.


Environment (conda list):

``` $ conda list # Name Version Build Channel av 0.4.2.dev0 pypi_0 pypi blas 1.0 mkl boost 1.67.0 py36_4 ceres-solver 1.14.0 h42bd5c8_2 conda-forge certifi 2019.3.9 py36_0 cython 0.29.6 pypi_0 pypi eigen 3.3.7 h74a9793_0 ffmpeg 4.1 h6538335_1002 conda-forge gflags 2.2.2 ha925a31_0 glew 2.0.0 he025d50_1002 conda-forge glfw 3.2.1 h2fa13f4_0 conda-forge glog 0.3.5 h6538335_1 hdf5 1.8.20 hac2f561_1 icc_rt 2019.0.0 h0cc432a_1 intel-openmp 2019.3 203 ipaddress 1.0.22 pypi_0 pypi jpeg 9b hb83a4c4_2 libblas 3.8.0 8_mkl conda-forge libboost 1.67.0 hfd51bdf_4 libcblas 3.8.0 8_mkl conda-forge libflang 5.0.0 h6538335_20180525 conda-forge liblapack 3.8.0 8_mkl conda-forge libopencv 3.4.2 h20b85fd_0 libpng 1.6.37 h2a8f88b_0 libtiff 4.0.10 hb898794_2 llvm-meta 5.0.0 0 conda-forge metis 5.1.0 h6538335_4 mkl 2019.3 203 mkl_fft 1.0.12 py36h14836fe_0 mkl_random 1.0.2 py36h343c172_0 msgpack 0.5.6 pypi_0 pypi ndsi 0.4 pypi_0 pypi numpy 1.16.2 pypi_0 pypi numpy-base 1.16.3 py36hc3f5095_0 openblas 0.3.6 h828a276_1 conda-forge opencv 3.4.2 py36h40b0b35_0 opencv-python 4.0.0.21 pypi_0 pypi openmp 5.0.0 vc14_1 conda-forge pillow 5.4.1 pypi_0 pypi pip 19.0.3 py36_0 psutil 5.5.1 pypi_0 pypi py-boost 1.67.0 py36h8300f20_4 py-opencv 3.4.2 py36hc319ecb_0 pyaudio 0.2.11 pypi_0 pypi pyglui 1.23 pypi_0 pypi pyopengl 3.1.0 pypi_0 pypi pyre 0.3.2 pypi_0 pypi pyrealsense2 2.21.0.776 pypi_0 pypi python 3.6.8 h9f7ef89_7 pyzmq 18.0.0 pypi_0 pypi scipy 1.2.1 pypi_0 pypi setuptools 40.8.0 py36_0 six 1.12.0 pypi_0 pypi sqlite 3.26.0 he774522_0 suitesparse 5.2.0 hdce8814_0 tbb 2019.4 h74a9793_0 torch 1.0.1 pypi_0 pypi torchvision 0.2.2 pypi_0 pypi tqdm 4.19.9 pypi_0 pypi uvc 0.13 pypi_0 pypi vc 14.1 h0510ff6_4 vs2015_runtime 14.15.26706 h3a45250_0 wheel 0.33.1 py36_0 win-inet-pton 1.1.0 pypi_0 pypi wincertstore 0.2 py36h7fe50ca_0 xz 5.2.4 h2fa13f4_4 zlib 1.2.11 h62dcd97_3 zstd 1.3.7 h508b16e_0 ```


Details about conda and system ( conda info ):

``` $ conda info active environment : pupil active env location : E:\SachsLab\Tools\Misc\condaenvs_win\pupil shell level : 2 user config file : C:\Users\Chad\.condarc populated config files : C:\Users\Chad\.condarc conda version : 4.6.14 conda-build version : not installed python version : 3.6.3.final.0 base environment : C:\ProgramData\Miniconda3 (writable) channel URLs : https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/win-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch https://conda.anaconda.org/conda-forge/win-64 https://conda.anaconda.org/conda-forge/noarch package cache : C:\ProgramData\Miniconda3\pkgs C:\Users\Chad\.conda\pkgs C:\Users\Chad\AppData\Local\conda\conda\pkgs envs directories : E:\SachsLab\Tools\Misc\condaenvs_win C:\ProgramData\Miniconda3\envs C:\Users\Chad\.conda\envs C:\Users\Chad\AppData\Local\conda\conda\envs platform : win-64 user-agent : conda/4.6.14 requests/2.18.4 CPython/3.6.3 Windows/10 Windows/10.0.17763 administrator : False netrc file : None offline mode : False ```
jschueller commented 5 years ago

seems ceres dll has everything it needs as ctypes can load it: https://github.com/conda-forge/ceres-solver-feedstock/pull/4

cboulay commented 5 years ago

Admittedly I don't use ctypes very much, but I seem to have a different result. I think your test environment in #4 installs some other components that I don't get with a simple conda environment. (I don't know what the -host section is doing in the meta.yaml file but maybe this is concealing the problem).

conda create -n ceres_test python=3.6 ceres-solver -c conda-forge

The following NEW packages will be INSTALLED:

  blas               pkgs/main/win-64::blas-1.0-mkl
  ceres-solver       conda-forge/win-64::ceres-solver-1.14.0-h42bd5c8_2
  certifi            conda-forge/win-64::certifi-2019.3.9-py36_0
  eigen              conda-forge/win-64::eigen-3.3.7-he980bc4_1000
  gflags             conda-forge/win-64::gflags-2.2.2-he025d50_1001
  glog               pkgs/main/win-64::glog-0.3.5-h6538335_1
  intel-openmp       pkgs/main/win-64::intel-openmp-2019.3-203
  libblas            conda-forge/win-64::libblas-3.8.0-8_mkl
  libcblas           conda-forge/win-64::libcblas-3.8.0-8_mkl
  liblapack          conda-forge/win-64::liblapack-3.8.0-8_mkl
  metis              conda-forge/win-64::metis-5.1.0-h6538335_1003
  mkl                pkgs/main/win-64::mkl-2019.3-203
  pip                conda-forge/win-64::pip-19.1.1-py36_0
  python             conda-forge/win-64::python-3.6.7-he025d50_1004
  setuptools         conda-forge/win-64::setuptools-41.0.1-py36_0
  suitesparse        pkgs/main/win-64::suitesparse-5.2.0-hdce8814_0
  tbb                conda-forge/win-64::tbb-2019.6-he980bc4_0
  vc                 pkgs/main/win-64::vc-14.1-h0510ff6_4
  vs2015_runtime     pkgs/main/win-64::vs2015_runtime-14.15.26706-h3a45250_4
  wheel              conda-forge/win-64::wheel-0.33.2-py36_0
  wincertstore       conda-forge/win-64::wincertstore-0.2-py36_1002

conda activate ceres_test python

import ctypes, os
dll_path = os.path.join(os.environ['CONDA_PREFIX'],'Library','bin','ceres.dll')
print("dll {} found".format(dll_path) if os.path.isfile(dll_path) else "dll not found")
ctypes.cdll[dll_path]
(ceres_test) C:\Users\Chad>python
Python 3.6.7 (default, Feb 28 2019, 07:28:18) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes, os
>>> dll_path = os.path.join(os.environ['CONDA_PREFIX'],'Library','bin','ceres.dll')
>>> print("dll {} found".format(dll_path) if os.path.isfile(dll_path) else "dll not found")
dll C:\Users\Chad\.conda\envs\ceres_test\Library\bin\ceres.dll found
>>> ctypes.cdll[dll_path]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Chad\.conda\envs\ceres_test\lib\ctypes\__init__.py", line 423, in __getitem__
    return getattr(self, name)
  File "C:\Users\Chad\.conda\envs\ceres_test\lib\ctypes\__init__.py", line 418, in __getattr__
    dll = self._dlltype(name)
  File "C:\Users\Chad\.conda\envs\ceres_test\lib\ctypes\__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found
cboulay commented 5 years ago

If you run DependenciesGUI from the Anaconda Prompt then it has the same PATH and it can help diagnose python's unhelpful OSError:

D:\Tools\Misc\Dependencies\DependenciesGUI.exe C:\Users\Chad\.conda\envs\ceres_test\Library\bin\ceres.dll

ceres_missing_dependencies

There are 4 dlls that cannot be found on the path, and indeed are not anywhere on my system.

jschueller commented 5 years ago

glog comes from defaults, this is unsupported : conda-forge must be the top-priority channel

cboulay commented 5 years ago

I get the same packages when conda-forge is my top-priority channel. On the conda-forge glog site here, there is no Windows entry.

(base) C:\Users\Chad>conda config --show channels
channels:
  - conda-forge
  - defaults
(ceres_test) C:\Users\Chad>conda install -c conda-forge glog
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: C:\Users\Chad\.conda\envs\ceres_test

  added / updated specs:
    - glog

The following NEW packages will be INSTALLED:

  gflags             conda-forge/win-64::gflags-2.2.2-he025d50_1001
  glog               pkgs/main/win-64::glog-0.3.5-h6538335_1
jschueller commented 5 years ago

oh, so glog shouldnt be pulled from ceres then

cboulay commented 5 years ago

Before you go too far down this rabbit hole, I'd like to emphasize that I have glog.dll on my path, and it isn't missing any dependencies. The missing dlls are spqr, cholmod, openblas, and cxsparse, and these are dependencies of ceres.dll, not of glog.dll.

jschueller commented 5 years ago

oh yes so it must be suitesparse then

jschueller commented 5 years ago

can you try the new revision of the package ?

cboulay commented 5 years ago

My conda command is still picking up the old one. I've never been on the bleeding edge of a conda release before. Can you please point me to instructions on how to use the new revision?

jschueller commented 5 years ago

maybe conda update --all ?

cboulay commented 5 years ago

It works! Thank you so much!

jid79 commented 4 years ago

hi . i've installed ceres pkg in python too. how do i continue from there. are there any examples of how to use it in python? i mean, what lib should i import... ?

Te1MinRyU commented 4 years ago

I also have the same problem on how to import ceres-solver after installation. No matter how I try, I still can't find the way to import this module. Is there anyone who knows how to use this library properly? Many thanks.

jschueller commented 4 years ago

maybe try with a clean installation from miniconda (and conda-forge as top priority channel)