search

BNLNPPS / esi-shell

Apache License 2.0
0 stars 0 forks source link

chore: use poetry for python dependencies #72

Closed plexoos closed 1 month ago

rahmans1 commented 2 months ago

Just curious, is it not possible to handle this through spack alone? Also is this just for pyvista https://packages.spack.io/package.html?name=py-pyvistaqt ?

plexoos commented 2 months ago

Yes, Spack can be used to install python packages. Looks like they have it https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-pyvista/package.py but I think there always be a lag between what Spack has and the releases in pypi https://pypi.org/project/pyvista/

rahmans1 commented 2 months ago

Yes, Spack can be used to install python packages. Looks like they have it https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-pyvista/package.py but I think there always be a lag between what Spack has and the releases in pypi https://pypi.org/project/pyvista/

We can request a version update or I can add myself as a maintainer for that spack package. It usually requires a small update like https://github.com/spack/spack/pull/41246 to have the version update if we require the most latest version.

plexoos commented 2 months ago

We can certainly consider Spack as a package manager for python... But I think for now a more pressing issue is that pyvista does not work on our GPU machine. Here I created an issue: #76 Could you please take a look?

buddhasystem commented 1 month ago

@plexoos so installing "poetry" and its paraphernalia is just another attempt to display pyvista on your Mac?

plexoos commented 1 month ago

No

plexoos commented 1 month ago

It's for non-Mac user like you

rahmans1 commented 1 month ago

I think the major python packages are available through spack. I am tempted to branch of beta and see how well that works instead of adding poetry as a separate dependency management tool.

https://packages.spack.io/package.html?name=py-ipython https://packages.spack.io/package.html?name=py-numpy

plexoos commented 1 month ago

I don't have enough experience with Spack managing python packages, and I don't know how it deals with virtual environments... Poetry has no problem detecting an existing venv by conda or pip, or create a new one, and all of this pretty much automatically.

Anyway, here is how I suggest to test the current PR to see if the results can be visualized:

./esi-shell -v PR-72
# then inside the shell do
~# opticks-full-prepare
~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh
rahmans1 commented 1 month ago

I don't have enough experience with Spack managing python packages, and I don't know how it deals with virtual environments... Poetry has no problem detecting an existing venv by conda or pip, or create a new one, and all of this pretty much automatically.

Anyway, here is how I suggest to test the current PR to see if the results can be visualized:

./esi-shell -v PR-72
# then inside the shell do
~# opticks-full-prepare
~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh

Do I need to grab the .toml file separately or set another environment variable?

[rahmans@npps0 ~]$ ./esi-shell -v PR-72
==> Using esi-shell image: ghcr.io/bnlnpps/esi-shell:PR-72
==> Running docker container 8f7f78ada833
==> Setting up opticks environment. Please wait...
==> Found existing /home/rahmans/.opticks. Will use it

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

root@8f7f78ada833:~# opticks-full-prepare
=== opticks-full-prepare : START Fri May 24 18:03:40 UTC 2024
=== opticks-prepare-installation : generating RNG seeds into installcache
2024-05-24 18:03:40.161 INFO  [149] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.161 INFO  [149] [main@9] QCurandState::desc cs.num 1000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin
2024-05-24 18:03:40.164 INFO  [150] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.164 INFO  [150] [main@9] QCurandState::desc cs.num 3000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin
2024-05-24 18:03:40.167 INFO  [151] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.167 INFO  [151] [main@9] QCurandState::desc cs.num 10000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin
=== opticks-full-prepare : DONE Fri May 24 18:03:40 UTC 2024
root@8f7f78ada833:~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh

Poetry could not find a pyproject.toml file in /home/rahmans or its parents
root@8f7f78ada833:~#
rahmans1 commented 1 month ago

I don't have enough experience with Spack managing python packages, and I don't know how it deals with virtual environments... Poetry has no problem detecting an existing venv by conda or pip, or create a new one, and all of this pretty much automatically. Anyway, here is how I suggest to test the current PR to see if the results can be visualized:

./esi-shell -v PR-72
# then inside the shell do
~# opticks-full-prepare
~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh

Do I need to grab the .toml file separately or set another environment variable?

[rahmans@npps0 ~]$ ./esi-shell -v PR-72
==> Using esi-shell image: ghcr.io/bnlnpps/esi-shell:PR-72
==> Running docker container 8f7f78ada833
==> Setting up opticks environment. Please wait...
==> Found existing /home/rahmans/.opticks. Will use it

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

root@8f7f78ada833:~# opticks-full-prepare
=== opticks-full-prepare : START Fri May 24 18:03:40 UTC 2024
=== opticks-prepare-installation : generating RNG seeds into installcache
2024-05-24 18:03:40.161 INFO  [149] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.161 INFO  [149] [main@9] QCurandState::desc cs.num 1000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin
2024-05-24 18:03:40.164 INFO  [150] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.164 INFO  [150] [main@9] QCurandState::desc cs.num 3000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin
2024-05-24 18:03:40.167 INFO  [151] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.167 INFO  [151] [main@9] QCurandState::desc cs.num 10000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin
=== opticks-full-prepare : DONE Fri May 24 18:03:40 UTC 2024
root@8f7f78ada833:~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh

Poetry could not find a pyproject.toml file in /home/rahmans or its parents
root@8f7f78ada833:~#

Anyways. I figured it out. It just doesn't know where the .toml file is automatically. I have to launch it from the right folder or have it in path.

rahmans1 commented 1 month ago 
This system does not appear to be running an xserver.
PyVista will likely segfault when rendering.

Try starting a virtual frame buffer with xvfb, or using
  ``pyvista.start_xvfb()``
  warnings.warn(
opticks/g4cx/tests/G4CXTest_raindrop.sh: line 189:   302 Segmentation fault      (core dumped) ${IPYTHON:-ipython} --pdb -i $script
opticks/g4cx/tests/G4CXTest_raindrop.sh : ana error with script G4CXTest_raindrop.py

Is this the error you get now? I guess we will just need to modify the plotting script so that it doesn't try to launch the headless window with xfvb which is not in our system.

buddhasystem commented 1 month ago

@rahmans1 If the host npps0 is not running the X server, we need to make it do so. Can you do it?

plexoos commented 1 month ago

Sakib, please use the updated esi-shell from this pr

On Fri, May 24, 2024, 2:15 PM Maxim Potekhin @.***> wrote:

@rahmans1 https://github.com/rahmans1 If the host npps0 is not running the X server, we need to make it do so. Can you do it?

— Reply to this email directly, view it on GitHub https://github.com/BNLNPPS/esi-shell/pull/72#issuecomment-2130116765, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGF6F7ZUE2VGQR77476UJDZD57TVAVCNFSM6AAAAABHY37CJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZQGEYTMNZWGU . You are receiving this because you were mentioned.Message ID: @.***>

rahmans1 commented 1 month ago

@rahmans1 If the host npps0 is not running the X server, we need to make it do so. Can you do it?

[rahmans@npps0 ~]$ startx xauth: file /home/rahmans/.serverauth.1541623 does not exist

/usr/libexec/Xorg.wrap: Only console users are allowed to run the X server

xinit: giving up xinit: unable to connect to X server: Connection refused xinit: server error Couldn't get a file descriptor referring to the console

rahmans1 commented 1 month ago

Sakib, please use the updated esi-shell from this pr …

I started with './esi-shell -v PR-72'. Is there something else that I am missing?

plexoos commented 1 month ago

I started with './esi-shell -v PR-72'. Is there something else that I am missing?

Please download the updated esi-shell script from the branch corresponding to this PR, like so:

curl -Os https://raw.githubusercontent.com/BNLNPPS/esi-shell/pr/poetry/esi-shell && chmod u+x esi-shell
rahmans1 commented 1 month ago

I started with './esi-shell -v PR-72'. Is there something else that I am missing?

Please download the updated esi-shell script from the branch corresponding to this PR, like so:

I get the same error after getting the updated esi-shell script

plexoos commented 1 month ago

You reported two errors. The updated esi-shell addressed the first one where it could not find a pyproject.toml. Now, you don't need to install with poetry by hand.

The second one about "This system does not appear to be running an xserver ... " is because you are running remotely and X windows cannot be opened locally. If you have a direct access to a GPU machine (NVidia of course) with X11, this error should not appear. Unfortunately, I don't have such machine to test but that is what I expect.

Maxim @buddhasystem it seems like you have a local NVidia GPU. Could you try the test I suggested above to see if we get any visuals from it?