robust & standard conda environment

[andrea-bistacchi]

After pulling the last version 5' ago,I got the unexpected error:

Basically vtkProjectPointsToPlane is not found in module vtkmodules.vtkFiltersPoints, but it is required here.

What version of VTK are you using? I have vtk 9.1.0 on Windows 10.

[gbene]

Yes I have encountered this error too when reinstalling the pzero environment. I resolved it by upgrading vtk to 9.2

[mcbaguetti]

Yes I have encountered this error too when reinstalling the pzero environment. I resolved it by upgrading vtk to 9.2

same for me

[mcbaguetti]

I noted some vtk imports are red, I was thinking to change some imports like in the img

[gbene]

Yes the vtk imports need to be rewritten. They all seem to work, but it is not the intended way to import the methods from the library. I have written a brief comment about this here: https://github.com/andrea-bistacchi/PZero/issues/8#issuecomment-1411636767.

[andrea-bistacchi]

yes!

[mcbaguetti]

about the issue Andrea described at the top, I ran some automated tests and I found conda couldn't solve the environment with vtk >= 9.2 on every OS because of dependencies conflicts. I solved this by installing vtk 9.2.* with pip and I assume @gbene did the same (or not? For checking this run conda list on the terminal and see if vtk has the channel pypi which is pip or conda-forge which is from conda).

This is relevant because installing vtk 9.2.* with pip allows running all the tests that directly or indirectly call vtkProjectPointsToPlane (actually are test_project_window and test_windows_factory).

If vtk 9.2.* is not installed we have two options:

1. we don't upload/run the tests that directly or indirectly call vtkProjectPointsToPlane
2. we delete vtkProjectPointsToPlane imports and their usage in the codebase

Of course, we have an additional option which is to solve the conda dependencies conflicts and I leave here attached the log of incompatibilities, at the end of the file you can see the vtk conflicts. I find this hard to solve because I think it's related more to how conda manages the environment.

[gbene]

I solved this by installing vtk 9.2.* with pip and I assume @gbene did the same

Mhhh, not really, I am on Linux and I didn't encounter any conflicts using conda-forge. On which OS(s) did the test fail? How do you determine a failure? I am asking because if it is determined by a time factor then it could be that conda is taking a lot of time in resolving the env (as we already discussed) and thus resulting in a failure.

I would rather not remove vtkProjectPointsToPlane since it is a useful function but maybe it is more important running tests on project_window and windows_factory since they are so central. In that case I can rewrite the code so that we don't use vtkProjectPointsToPlane but I need some time. @andrea-bistacchi what do you think?

Maybe it's time to move to pip? It will be a bit of a headache I think.

[andrea-bistacchi]

Hi, I guess @luca-penasa's answer would be "move to pip".

However, in the past I found out that the order you use to install conda modules really matters. If you first install the more problematic modules (vtk in this case) and then the others, instead of trying to install all at once, creating an environment tends to be much smoother.

I'm testing this in these days since I have a new laptop and I'll let you know.

BTW: removing vtkProjectPointsToPlane is not an option.

[mcbaguetti]

On which OS(s) did the test fail?

the failures are both on my local Windows machines (both 10 and 11) and on GitHub Actions which runs Ubuntu, macOS, and Windows. I think it's not determined by a time factor because it's failing after ~5 minutes, sometimes less based on the OS. Can you share how you update vtk?

How do you determine a failure?

Basically, on GitHub Actions we can run a set of actions on virtual machines (for ex., for testing purposes). So a failure means one of the actions performed gave an error and interrupted the action itself. This leads to stopping all the workflows of actions, you can see here some examples of correct/stopped/failed workflows. Inside each workflow run you can see one or more OS (it depends on the try), and inside each OS you can see all the steps the workflow runs.

The main failures here are these:

1. when I don't install vtk 9.2 (so there is 9.1 by default), it fails the step Test with Pytest because of the import vtkProjectPointsToPlane (example here). The solution here would be to delete or comment the two tests I mentioned before.
2. I try to install vtk 9.2 within conda environment and it fails on the step Creating environment from .yml which is responsible for installing all the dependencies from the environment.yml (an example could be this).

I would rather not remove vtkProjectPointsToPlane since it is a useful function

I think the same, if it's important we can leave it!

The solution I found for GitHub Actions I think is temporary, as I explained before, it installs vtk 9.2 with pip only on GitHub Actions but without modifying the environment.yml, and this is good because allows running all the tests and not changing the dependencies.

[andrea-bistacchi]

Hi, this agrees with my experience on local Windows 10 machines:

• vtk 9.1 is missing vtkProjectPointsToPlane

• creating the environment can take a lot of time with vtk 9.2, much more than the 5 minutes allowed by GitHub Actions

Is there a way to create the environment incrementally in GitHub Actions, adding vtk (an possibly other problematic libraries) first, and the the others?

[mcbaguetti]

Is there a way to create the environment incrementally in GitHub Actions, adding vtk (an possibly other problematic libraries) first, and the the others?

Incrementally it works but only with pip installing vtk after it's created the conda environment.

[mcbaguetti]

I also leave here the wiki of the project, I created in there some small guides to testing, actions, and auto-documentation. I will continue to add some pages in the next few days to help understand better the project. Feel free to add all the pages you want or to modify or improve them.

[andrea-bistacchi]

Hi, from this article, it looks like there is a possibility to create a conda virtual environment incrementally, and that this approach simplifies the workload for conda.

Particularly here they first create the environment with a given Python release and other libraries, then add some more.

I don't know if this makes sense in GitHub Actions...

[mcbaguetti]

Particularly here

I will try that option.

Until now I tried to create incrementally using before conda install vtk>9.2 and then adding the other dependencies but it didn't work.

[luca-penasa]

On linux to make the environment solvable I updated the version of qt to 5.15:

  - pyqt=5.15
  - vtk=9.2

vtk was refusing to install due to the fact that vtk-9.2 would require qt >=5.15.8, as far as I got

on linux, with micromamba.

I am not sure being so strict on pyqt version is needed anyway... maybe having just <6 would be enough?

Luca

[andrea-bistacchi]

Hi, I don't know if this is relevant, but I had a look at environment.yaml files of PyVista and pyvistaqt, and they are pretty different. The Python requirement is different (< 3.9 vs > 3.8), and pyvistaqt uses pip for vtk and qt (probably the same problem as our...) while PyVista uses conda (just for vtk), but nobody specifies a vtk or qt release.

Maybe some conflicts or difficulties in handling dependencies arise from this?

One test we can run is to remove pyvista from our requirements (it will be implied by pyvistaqt), specify vtk>=9.2 (we need it as discussed above), and then we can try with or without an explicit requirement for qt (it should be implicit with pyvistaqt, but maybe we need a specific version).

Let us know what you think!

[andrea-bistacchi]

Other discussions suggest either to use mamba instead of conda and/or to pre-cache the libraries. This could make sense anyway.

[andrea-bistacchi]

Maybe we should change the title of this issue.

[mcbaguetti]

Hi all, I noted now deleting pyvista and pyvista-qt doesn't solve too the environment for GitHub Actions. But if we require pyqt=5.15, it works with vtk >=2, pyvista, pyvista-qt and all the other dependencies.

[andrea-bistacchi]

Good!

Is the environment.yml file up to date? I noticed it still includes vtk=9.1, which solves come incompatibilities but excludes vtkProjectPointsToPlane().

[mcbaguetti]

Is the environment.yml file up to date?

Not yet in the main branch, I wanted to test it on the auto-testing branch to see if it could have worked! Today I will merge or update all the progresses on this side on the main branch!

[andrea-bistacchi]

Thanks I found it!

[andrea-bistacchi]

Hi, I changed the title for obvious reasons.

[andrea-bistacchi]

I still cannot install easily from environment.yml, so I'm creating the environment step-by-step and documenting all the process here. I hope this helps!

conda create -n pzero python=3.8

I used Python 3.8 since it seems more compatible with some packages below.

conda activate pzero
conda config --env --set channel_priority strict
conda config --env --add channels loop3d
conda config --env --add channels conda-forge
conda install -c conda-forge geopandas=0.11

Geopandas is a real bottleneck so I install it first of all. Release 0.11 should be the last one compatible with Shapely 1.8. By installing geopandas also fiona, gdal, matplotlib-base, numpy, scipy, scikit-learn, pandas=2.0, proj, pyproj, shapely=1.8 and many others are installed.

conda install vtk=9.2

VTK installs many libraries including qt-main=5.15.8.

conda install pyqt=5.15
conda install pyvistaqt

This works very well and the latter also installs imageio, pyvista=0.39, and qtpy.

conda install -c conda-forge rasterio vedo
conda install -c loop3d loopstructural

This works well and the latter also installs scikit-image.

conda install -c conda-forge cmocean colorcet mplstereonet seaborn laspy ezdxf lxml openpyxl pythreejs

This works very well.

conda install -c conda-forge xarray=2023.1.0

This caused some problem. I selected this particular release based on the output of conda search xarray --info, since it is the last release compatible with Python 3.8. The installation downgraded Pandas (installed with geopandas) from 2.0.1 to to 1.5.3. I hope this is not a problem...

conda install pytest
conda install -c conda-forge sphinx sphinx_rtd_theme
conda install -c conda-forge pyinstaller

This works with channels as specified above.

Note that for pytest and spinx the anaconda channel is suggested on the conda web site, but this does not work since using it causes conflicts with many other packages. The sphinx_rtd_theme must be installed at the same time as sphinx since it is not compatible with the latest release of the latter, as shown after running conda search sphinx_rtd_theme --info.

BTW, I tried installing mamba but it was taking a lot of time so I gave up. Pre-caching the libraries could be good idea, but I guess this is something to be tested by @mcbaguetti

[andrea-bistacchi]

The step-by-step installation above resulted, on Window 11 64 bit, in the following packages being explicitly installed from the conda-forge and loop3d channels with strict channel priority:

python=3.8.16 geopandas=0.11.1 vtk=9.2.6 pyqt=5.15.7 pyvistaqt=0.10.0 rasterio=1.3.6 vedo=2023.4.4 loopstructural=v1.5.10 cmocean=3.0.3 colorcet=3.0.1 mplstereonet=0.6.2 seaborn=0.12.2 laspy=2.3.0 ezdxf=1.0.3 lxml=4.9.2 openpyxl=3.1.2 pythreejs=2.4.2 xarray=2023.1.0 pytest=7.3.1 sphinx=5.3.0 sphinx_rtd_theme=1.2.0 pyinstaller=5.10.1

I added all these release specifications to environment.yml in the env_testing branch and I have verified that this speeds installation up very much, at least on Windows 11.

[gbene]

I'll try it on my linux machine

[gbene]

Ok so it worked on my machine but after a couple of strange crashes (segmentation fault).

I think it is a Linux problem. I hope that the linux machines that will run the automatic tests won't encounter the same problem.

[andrea-bistacchi]

You can also compare the release of each library with the one you had in Linux with the previous environment.yml file (still in master branch), with Python 3.9 and just a few release specs.

If something is newer in your version, we can try using the same in Windows as well.

[mcbaguetti]

I tested with Actions and only on the virtual MacOS it doesn't find loopstructural v1.5.10**

[andrea-bistacchi]

From the loop3d channel it looks like the most recent loopstructural release for Mac is 1.4.4. I think we can use that in order to be able to compile for all platforms. Please test also this solution.

[mcbaguetti]

From the loop3d channel it looks like the most recent loopstructural release for Mac is 1.4.4. I think we can use that in order to be able to compile for all platforms. Please test also this solution.

It doesn't work for Windows with 1.4.4, the alternative could be to use only v1.4.4 for macOS and the other one for Windows and Linux. This could be accomplished with the environment.yml without LoopStructural and then added manually with conda install ... or we can create two different environments.

[andrea-bistacchi]

Hi, so far as the functions we use are concerned, it should not matter. Let's use the simpler solution!

[andrea-bistacchi]

Hi, according to a quick Google search it is possible to run different actions based on the os. In this case we could point to an environment_macos different from the standard one that works with windows and linux. This could be an option even if in general I would like something more homogeneous.

[mcbaguetti]

I created an envs folder that contains the two environments and this works as expected on all OS

[andrea-bistacchi]

Great news!

[andrea-bistacchi]

Great update! I switched to the new libmamaba package solver, that is very much faster and does not give all problems listed above.

conda update -n base conda
conda install -n base conda-libmamba-solver
conda config --set solver libmamba

Installing in the base environment is all you need.

[andrea-bistacchi]

With this update I was able to update the pzero environment with PySide6 (the LGPL version of PyQt6), QtPy (abstraction library used to manage PySide vs- PyQt transparently) and all the requirements above.

After removing the old pzero environment with:

conda remove --name pzero --all

this is what I did exactly:

conda create -n pzero python=3.8
conda activate pzero
conda config --env --set channel_priority strict
conda config --env --add channels loop3d
conda config --env --add channels conda-forge
conda install -c conda-forge shapely=1.8 geopandas
conda install -c conda-forge pyside6=6.5
conda install -c conda-forge qtpy
conda install vtk>=9.2
conda install pyvistaqt
conda install -c conda-forge rasterio vedo
conda install -c loop3d loopstructural
conda install -c conda-forge cmocean colorcet mplstereonet seaborn laspy ezdxf lxml openpyxl pythreejs
conda install -c conda-forge xarray=2023.1.0
conda install pytest
conda install -c conda-forge sphinx sphinx_rtd_theme
conda install -c conda-forge pyinstaller

The new std-environment.yml looks like this:

name: pzero
channels:
  - conda-forge
  - loop3d
  - defaults
dependencies:
  - python=3.8.17
  - geopandas=0.13.2
  - shapely=2.0.1
  - pyside6=6.5.1
  - qtpy=2.3.1
  - vtk=9.2.6
  - pyvistaqt=0.10.0
  - rasterio=1.3.7
  - vedo=2023.4.4
  - loopstructural=v1.5.10
  - cmocean=3.0.3
  - colorcet=3.0.1
  - ezdxf=1.0.3
  - laspy=2.3.0
  - lxml=4.9.2
  - mplstereonet=0.6.2
  - openpyxl=3.1.2
  - pythreejs=2.4.2
  - seaborn=0.12.2
  - xarray=2023.1.0
  - pytest=7.3.2
  - sphinx=6.2.1
  - sphinx_rtd_theme=1.2.2
  - pyinstaller=5.12.0

[andrea-bistacchi]

I'm going back to PyQt5. PySide is not completely compatible with our code and PyQt6 is not available on conda.

All other packages are the same as above, but I now install all packages at the same time specifying only shapely=1.8 aand vtk>=9.2. This prevents Shapely from being updated to 2.0 and is very fast thanks to the new libmamaba package solver (it allowed testing all this in minutes).

conda activate base
conda remove --name pzero --all
conda create -n pzero python=3.8
conda activate pzero
conda config --env --set channel_priority strict
conda config --env --add channels loop3d
conda config --env --add channels conda-forge 
conda install pyqt qtpy shapely=1.8 geopandas vtk>=9.2 pyvistaqt rasterio vedo xarray cmocean colorcet mplstereonet seaborn laspy ezdxf lxml openpyxl pythreejs pytest sphinx sphinx_rtd_theme pyinstaller loopstructural

I have exported the std-environment.yml with --from-history and a complete environment.yml without this option. We could check if the latter now works on all platforms.

conda env export --from-history > "D:\REPOS\PZero\envs\std-environment.yml"
conda env export > "D:\REPOS\PZero\envs\environment.yml"

[luca-penasa]

I can also confirm on Linux (arch) I was able to set up a running environment by using std-environment.yml using

micromamba env create -n pzero -f envs/std-environment.yml

as a sidenote for @andrea-bistacchi & @gbene:

with the versions of packages in that environment, the issue with QSpinBox.setValue requiring an int, but receiving a float, does not show up.

I tested also the creation of the env with python 3.9, 3.10. the error shows up starting from version 3.10. I'd suggest modifying them to be QDoubleSpinBox anyway as I believe they are intended to hold a floating point.

probably some automated casting was dropped at that point. Might be related to this:

Builtin and extension functions that take integer arguments no longer accept Decimals, Fractions and other objects that can be converted to integers only with a loss (e.g. that have the int() method but do not have the index() method). (Contributed by Serhiy Storchaka in bpo-37999.)

Not the right place, but are we sure we still want to target python 3.8? 3.10 and 3.11 are becoming the default now and introduced many interesting features.

Luca

[andrea-bistacchi]

I agree! Maybe @mcbaguetti can test it on github actions and we will act based on that.

[mcbaguetti]

the environment with python 3.10 work fine for windows and ubuntu, not macos which would need a loopstructural version change

Could not solve for environment specs The following packages are incompatible ├─ loopstructural v1.4.4* is installable with the potential options │ ├─ loopstructural v1.4.4 would require │ │ └─ python >=3.8,<3.9.0a0 , which can be installed; │ ├─ loopstructural v1.4.4 would require │ │ └─ python >=3.7,<3.8.0a0 , which can be installed; │ └─ loopstructural v1.4.4 would require │ ├─ python >=3.9,<3.10.0a0 , which can be installed; │ └─ python_abi 3.9. *_cp39, which can be installed;

[andrea-bistacchi]

OK, if the problem with loop is the same as in Python 3.8, we can move on to 3.10.

[andrea-bistacchi]

Hi, @gbene is updating QSpinBox to QDoubleSpinBox, then we are free to move on to 3.10.

[gbene]

hi! I've updated the legend to doublespinbox. I wanted to test it on python 3.10 but I am having the same old difficulties with the enviroment (also in the base env). I think that I need to delete everything and install conda from scratch.

[andrea-bistacchi]

already performed a push? I can test it

[gbene]

yup, along with the View class reformatting start

[andrea-bistacchi]

Hi, I added blackd (automatic Python code formatting) to the standard pzero environment. This allows finding the blackd executable used by BlackConnect plugin for PyCharm. To generate the environment use:

conda update -n base -c conda-forge conda
conda activate base
conda remove --name pzero --all
conda create -n pzero python=3.8
conda activate pzero
conda config --env --set channel_priority strict
conda config --env --add channels loop3d
conda config --env --add channels conda-forge 
conda install pyqt qtpy shapely=1.8 geopandas vtk>=9.2 pyvistaqt rasterio vedo xarray cmocean colorcet mplstereonet seaborn laspy ezdxf lxml openpyxl pythreejs pytest sphinx sphinx_rtd_theme pyinstaller loopstructural blackd

The resulting .yml files have been pushed to the repo:

conda env export --from-history > "D:\REPOS\PZero\envs\std-environment.yml"
conda env export > "D:\REPOS\PZero\envs\environment.yml"

[andrea-bistacchi]

Since Conda 23.10.0, libmamaba is the default package solver, as discussed here.

This closes this issue.

The last environment tested here on July 20th 2023 is now the standard PZero environment.

Se more details on the environment in the Wiki.