Closed PaulaMihalcea closed 3 years ago
The current installation guide is unclear about the actual usage of the psbody-mesh package after building it, as well as impossible to execute on machines without sudo permissions.
Thanks. The only difficult part is about Boost, and the easiest way to install it on Linux/OSX is the way described in the document. I personally never use Conda so I would not make it a requirement for installing the mesh package. Rather I would help the user get Boost installed if sudo
rights are lacking (very well explained on Boost website).
Conda is however a widely used system, any instructions to get mesh working with conda are valuable. I would prefer having a dedicated section such as "install with conda" instead of replacing the existing one. However the hacky solution is a no-go for me.
If you look at the make all
command, it is only about 2 instructions:
pip install --upgrade -r requirements.txt
pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir .
I am not familiar with conda
but I guess we can emulate those 2 instructions with vanilla conda
commands that do not require the cloning and renaming steps?
The only difficult part is about Boost, and the easiest way to install it on Linux/OSX is the way described in the document. I personally never use Conda so I would not make it a requirement for installing the mesh package. Rather I would help the user get Boost installed if
sudo
rights are lacking (very well explained on Boost website).
I must have totally missed the part in the Boost documentation where they say how to install it without sudo
rights, which is the reason why I used Conda. My bad. At this point though I am leaving this for posterity, as I am in no way knowledgeable about Boost.
Conda is however a widely used system, any instructions to get mesh working with conda are valuable. I would prefer having a dedicated section such as "install with conda" instead of replacing the existing one.
I must admit that I was not sure about replacing the original instructions with the Conda version, but then I assumed that Conda was widespread and did it anyway. Since you are proof that not everybody uses it, it is only fair that we leave both alternatives in the guide.
However the hacky solution is a no-go for me.
Well, it is a no-go for me, too, but is the only way that works. Please believe me when I say that I have tried a lot of solutions. I absolutely do not exclude the possibility that I missed some vanilla conda
instruction for doing it in a nicer way, but then again, I did not find anything else.
If you look at the
make all
command, it is only about 2 instructions:pip install --upgrade -r requirements.txt pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir .
I am not familiar with
conda
but I guess we can emulate those 2 instructions with vanillaconda
commands that do not require the cloning and renaming steps?
I do not get it, the cloning is necessary to get the files needed for building the psbody-mesh
package from source; how could someone compile it without downloading the files, and what better way of doing it other than cloning into the official repository? In the past I stumbled upon other libraries that needed compiling from source, and they basically did the same thing (an example is PyTorch, see here).
The renaming step can be skipped by executing only git clone https://github.com/MPI-IS/mesh ~/anaconda3/envs/my_venv/lib/python3.8/site-packages/psbody
at step 4, which basically does the same thing (clone the psbody-mesh
source code in a folder called psbody
in the site-packages
folder), but in one command.
I would like to specify that the renaming part is necessary because the default name of the psbody-mesh
repo is mesh
, while the library is supposed to be imported in Python as import psbody.mesh
(at least this is how I found it in scripts that used it, like here or here); if you do not rename it, then the import statements must be changed to import mesh.mesh
(as the parent folder is mesh
), which does not seem to be the way that it was intended to be imported in.
As for the cd
part, I found that it can be avoided by specifying the psbody-mesh
folder directly in the make all
command, so step 5 becomes BOOST_INCLUDE_DIRS=~/anaconda3/envs/my_venv/include/ make -C ~/anaconda3/envs/my_venv/lib/python3.8/site-packages/psbody all
. Personally, I prefer keeping the cd
part for the sake of clarity, but I just tested it and either way works just fine.
There might be a way of building psbody-mesh
from source using Conda commands, see Building conda packages from scratch, and then installing it from the generated package. Personally, I did not even think about searching for something like this; everything I know about Conda I actually learnt while trying to get psbody-mesh
to work, so many aspects are new to me, too. This solution however looks more like something for the developers than for the final user - it even says how to upload packages to Anaconda for everyone to use. At the moment I have to meet a tight dealine on a project I am working on (indeed, the one where all this started for me), so I am not in a position to further investigate this approach, as it goes well beyond my knowledge about Conda. It does look promising, but considering that the building requires Boost, there might be some complications along the way, and I am not sure if it would be worth examining if the goal would only be creating a more elegant solution.
Took me a while to understand what you meant. If all what make all
does consists of running two pip
commands, then it should be as easy as running them in the activated conda
environment instead of using make all
, so after cloning in any folder, step 5 would literally become:
$ pip install --upgrade -r requirements.txt
$ pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir
However, I just tried it and it does not work; Python throws ModuleNotFoundError: No module named 'psbody'
.
Since when using Conda it is recommended to actually stick with conda
and not use pip
(some truly valuable advice), I tried replacing the first pip
command with conda
, so $ pip install --upgrade -r requirements.txt
translates to:
$ conda install --file requirements.txt
$ conda update --file requirements.txt
Too bad that the very first command (conda install
) throws an error, too, namely a PackagesNotFoundError
for opencv-python
(which looks like it should be installed separately), so I could not continue. One would probably need to add some new channels to the default Conda channels list by using conda config --append channels CHANNEL_NAME
, but at this point the requirements.txt
kinda loses meaning since it would be easier to manually install each required package instead of manually adding the correct channel for each/most of them before running the conda install
command.
After these new findings I am still convinced that the simplest solution is the hacky, no-go way, however ugly it might seem; I remain open to further suggestions, though, possibly from someone more experienced with Conda than us.
I updated the committed README.md file and restored the original installation guide by separating it from the Conda version; I also simplified steps 4 and 5 with the mentioned commands (those that still work).
There are various things here, but I think we start getting on the right path:
python setup.py install
step is performed. This installs the package properly into the virtual environment. You should not be needing to clone to site-package
at all ever.setup.py
called (if my memory does not fail me) meta package
or something. This means that the layout of the source and the layout of the installed package are different. We are not sticking to the source layout.requirements.txt
as is does not work, I think we can just find the right equivalent conda packages. opencv
should be available. I am all fine with not using a requirement file and having that step in the README you are editing (just like the pytorch installation step).pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir .
is no more that some python setup.py install
with additional options and a clean temporary (and isolated) folder for compiling the stuff. The only specific part is the option that passes the location where Boost is (not needed if Boost installed system-wide) and possibly some setuptools
specific stuff that I am pretty sure conda
is compatible with. That to say that if we can pass the requirement part, maybe this part will solve by itself. There are various things here, but I think we start getting on the right path
Yes, we are. Keep reading!
* you are cloning directly inside the virtual environment. If you look again at the pytorch installation, in particular [this section](https://github.com/pytorch/pytorch#install-pytorch), after installing various common conda package, a proper `python setup.py install` step is performed. This installs the package properly into the virtual environment. You should not be needing to clone to `site-package` at all ever.
Oh, you meant cloning into that specific folder! I thought you meant that the user was not supposed to ever clone into the repo at all. Alright, it makes sense now; I agree that it is not a good practice, I only did it this way to avoid moving the downloaded folder.
* the renaming: this is a functionality of `setup.py` called (if my memory does not fail me) `meta package` or something. This means that the layout of the source and the layout of the installed package are different. We are not sticking to the source layout.
I know nothing about this stuff, but earlier I happened to see this line in the psbody-mesh
setup.py
file, and so I think I understand what you mean.
* If the `requirements.txt` as is does not work, I think we can just find the right equivalent conda packages. `opencv` should be available. I am all fine with not using a requirement file and having that step in the README you are editing (just like the pytorch installation step).
Yeah, opencv
should work just fine and is available from more conventional Conda channels like conda-forge
. While I do like the concept of requirements files, I can say from experience that tons of problems arise from them (they contain the wrong versions of the packages, or contain no versions at all, stuff like this), so installing them manually can actually allow a more inexperienced user to understand what he/she is doing (e.g. when I first started the project I am working on, I had no idea what these requirements.txt
files had inside, and I could not figure out why I kept getting errors during package installation - then I realized they were nothing more than a list of packages, and proceeded with their manual installation).
* what `pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir .` is no more that some `python setup.py install` with additional options and a clean temporary (and isolated) folder for compiling the stuff. The only specific part is the option that passes the location where Boost is (not needed if Boost installed system-wide) and possibly some `setuptools` specific stuff that I am pretty sure `conda` is compatible with. That to say that if we can pass the requirement part, maybe this part will solve by itself.
This is where the good news come in play. I did some more researching after writing my previous comment and realized that in this case using pip
in a Conda environment is the right way of proceeding. At this point I thought that not having run python setup.py install
when trying to translate the make all
command could have been the culprit. I then tried adding this command after pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir
(I did not know that they did the same thing) and it worked, so I thought I solved the problem.
Just in case, I created another virtual environment and tried to do it again, and surprise! it did not work. It took me another 30 minutes of extensive research to notice a small detail in your pip install
command, something which looked like a typo, but was definitely intentional: the final dot .
. Yep, I omitted the dot the first time I tried the pip
commands, and missed the error statement in the countless output lines that flooded the terminal... and that is why pip
did not work.
So in conclusion, by completely skipping steps 3, 4 and 5 and executing the following instead, the import statement works correctly:
$ git clone https://github.com/MPI-IS/mesh
$ pip install --upgrade -r mesh/requirements.txt
$ pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir mesh/.
And an even nicer thing here is that you can safely delete the mesh
folder cloned with git
: the imports will still work, since the installation has been executed in a correct way. I just edited the README.md file accordingly.
All looks good to me @PaulaMihalcea !
Coming back with a quick tip:
if for some reason the second pip
command at step 4 does not work (pip install --no-deps --install-option="--boost-location=$$BOOST_INCLUDE_DIRS" --verbose --no-cache-dir mesh/.
), try replacing $$BOOST_INCLUDE_DIRS
with the path to the Boost installation of the virtual environment you are using, which should be something like ~/anaconda3/envs/my_venv/include/
; for example:
pip install --no-deps --install-option="--boost-location=~/anaconda3/envs/my_venv/include/" --verbose --no-cache-dir mesh/.
Remember to also replace my_venv
with the name of your virtual environment, if different.
Hi @PaulaMihalcea and @raffienficiaud , very sorry for my slow response. Thank you @PaulaMihalcea for your PR, and thank you very much @raffienficiaud for your comments and your review.
Yes, that looks good to me too. Merging!
The current installation guide is unclear about the actual usage of the psbody-mesh package after building it, as well as impossible to execute on machines without sudo permissions.
The installation part of the documentation has been thus rewritten in order to make it more clear and complete, as well as sudo-free (by using Anaconda).
The parts with the download in the site-packages folder and the renaming of the mesh folder to psbody is what actually makes using this package possible, and this rectified version of the guide contains the cleanest possible way of doing it - the alternatives being copying the mesh folder after compilation in the folders of the projects that use it (bad) and/or editing the import lines in the relative scripts (even worse), as well as the PYTHONPATH variable.