search

scipion-em / scipion-pyworkflow

Underlying pyworkflow module for the Scipion framework
GNU General Public License v3.0
6 stars 5 forks source link

picking wizards not working #129

Closed azazellochg closed 4 years ago

azazellochg commented 4 years ago

again java/xmippViz problems. Why there is bash?

java -Xmx2g -Djava.library.path=/home/azazello/soft/scipion3/software/em/xmipp/lib -Dplugins.dir=/home/azazello/soft/scipion3/software/em/xmipp/bindings/java/imagej/plugins -cp lib/*:/home/azazello/soft/scipion3/software/em/xmipp/bindings/java/imagej/*:/home/azazello/soft/scipion3/software/em/xmipp/bindings/java/lib/* xmipp.viewer.particlepicker.training.SupervisedPickerRunner --input Runs/000002_ProtImportMicrographs/micrographs.sqlite --output Tmp/2.outputMicrographs --mode Automatic --classifier Tmp/2.outputMicrographs/picker.conf
Preprocess command: NULL

Autopick command: 
python /home/azazello/soft/scipion-em-plugins/scipion-em-gautomatch/gautomatch/run_gautomatch.py Runs/000002_ProtImportMicrographs/extra/KLH_Dataset_I_Test_0001.mrc Tmp/2.outputMicrographs/references.mrcs Tmp/2.outputMicrographs  --apixM 4.40 --ang_step 5 --diameter 500 --lp 30 --hp 1000 --gid 0 --apixT 4.40 --speed 2 --boxsize 150 --lsigma_cutoff 2.00 --lsigma_D 100 --lave_max 1.00 --lave_min -1.00 --lave_D 100 --detect_ice 1 --T_norm_type 1 --do_bandpass 1 --cc_cutoff 0.18 --min_dist 300

Autopick output: 
/bin/bash: /home/azazello/soft/miniconda3/envs/.scipion3env/lib/libtinfo.so.6: no version information available (required by /bin/bash)
/bin/bash: python: command not found

Convert command: 
emconvert --coordinates --from gautomatch --to xmipp --input  Runs/000002_ProtImportMicrographs/micrographs.sqlite --output Tmp/2.outputMicrographs

Convert output: 
/bin/bash: /home/azazello/soft/miniconda3/envs/.scipion3env/lib/libtinfo.so.6: no version information available (required by /bin/bash)
/bin/bash: emconvert: command not found

PS. Running manually above commands via "scipion run ..." works

pconesa commented 4 years ago

In my machine code is working: image

It must be something different in our machines. So I'm not able to reproduce this.

What I can see is that the bash is coming from ShowJ.

I'm removing the "bash" "-c" from https://github.com/I2PC/xmippViz/blob/release-20.07/java/src/xmipp/utils/XmippWindowUtil.java#L300

And it does not work. This bash -c stuff is being there since at least scipion1.2: https://github.com/I2PC/scipion/blob/v1.2/software/em/xmipp/java/src/xmipp/utils/XmippWindowUtil.java

So there must be something else we are missing here.

pconesa commented 4 years ago

I'm on a virtualenv installation, that is one difference.

pconesa commented 4 years ago

I think it could be related with the environment setup defined by xmipp.

What do you have in xmippEnv.json, I have it beside the xmipp.conf file?

pconesa commented 4 years ago

I'm getting something similar in a conda installation

pconesa commented 4 years ago

I think I got it. Could you please make a quick test:

At https://github.com/I2PC/scipion-em-xmipp/blob/release-20.07/xmipp3/__init__.py#L80

replace it by

environ.addLibrary(Config.getPythonLibFolder(),position=pos)

You'll also need the devel version fo the plugin (released one is not recent?)

azazellochg commented 4 years ago

I have modified /home/azazello/soft/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/xmipp3/init.py as you suggested, didnt help. xmippEnv.json below:

{
    "LD_LIBRARY_PATH": "/usr/lib/x86_64-linux-gnu/stubs:/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/home/azazello/soft/miniconda3/envs/.scipion3env/lib",
    "PATH": "/usr/lib/jvm/java-11-openjdk-amd64/bin",
    "CUDA": "True"
}

xmipp.bashrc:

# This script is valid for bash and zsh

export XMIPP_HOME=/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build
export XMIPP_SRC=/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/src
export PATH=/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/bin:$PATH
export LD_LIBRARY_PATH=/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/lib:/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/bindings/python::/home/azazello/soft/miniconda3/envs/.scipion3env/lib:$LD_LIBRARY_PATH
export PYTHONPATH=/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/bindings/python:/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/pylib:$PYTHONPATH

alias x='xmipp'
alias xsj='xmipp_showj'
alias xio='xmipp_image_operate'
alias xis='xmipp_image_statistics'
alias xih='xmipp_image_header'
alias xmu='xmipp_metadata_utilities'

scipion-em-xmipp is 20.7b1

pconesa commented 4 years ago

I had my hopes in that change.

Can you remove (temporaly) /home/azazello/soft/miniconda3/envs/.scipion3env/lib from the LD_LIBRARY_PATH in the xmippEnv.json and try the picker.

azazellochg commented 4 years ago

nope, still same output as in 1st message

pconesa commented 4 years ago

Thanks.

Can you add here: https://github.com/I2PC/scipion-em-xmipp/blob/release-20.07/xmipp3/__init__.py#L81

a print line:

print("LD_LIBRARY_PATH: %s" % environ["LD_LIBRARY_PATH"])

And send output, please.

azazellochg commented 4 years ago 
Scipion v3.0 () devel

------------- refreshing ---------- 
  open files:  2
    - /home/azazello/ScipionUserData/projects/TestGautomatchAutomaticPicking/project.sqlite, 5
    - /home/azazello/ScipionUserData/projects/TestGautomatchAutomaticPicking/settings.sqlite, 6
  memory percent:  1.5041950875023535
------------- refreshing ---------- 
  open files:  2
    - /home/azazello/ScipionUserData/projects/TestGautomatchAutomaticPicking/project.sqlite, 5
    - /home/azazello/ScipionUserData/projects/TestGautomatchAutomaticPicking/settings.sqlite, 6
  memory percent:  1.5187488996121508
LD_LIBRARY_PATH: /usr/lib/x86_64-linux-gnu/stubs:/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/home/azazello/soft/scipion3/software/em/xmipp/lib:/home/azazello/soft/miniconda3/envs/.scipion3env/lib
java -Xmx2g -Djava.library.path=/home/azazello/soft/scipion3/software/em/xmipp/lib -Dplugins.dir=/home/azazello/soft/scipion3/software/em/xmipp/bindings/java/imagej/plugins -cp lib/*:/home/azazello/soft/scipion3/software/em/xmipp/bindings/java/imagej/*:/home/azazello/soft/scipion3/software/em/xmipp/bindings/java/lib/* xmipp.viewer.particlepicker.training.SupervisedPickerRunner --input Runs/000002_ProtImportMicrographs/micrographs.sqlite --output Tmp/2.outputMicrographs --mode Automatic --classifier Tmp/2.outputMicrographs/picker.conf
Preprocess command: NULL

Autopick command: 
python /home/azazello/soft/scipion-em-plugins/scipion-em-gautomatch/gautomatch/run_gautomatch.py Runs/000002_ProtImportMicrographs/extra/KLH_Dataset_I_Test_0001.mrc Tmp/2.outputMicrographs/references.mrcs Tmp/2.outputMicrographs  --apixM 4.40 --ang_step 5 --diameter 500 --lp 30 --hp 1000 --gid 0 --apixT 4.40 --speed 2 --boxsize 150 --lsigma_cutoff 2.00 --lsigma_D 100 --lave_max 1.00 --lave_min -1.00 --lave_D 100 --detect_ice 1 --T_norm_type 1 --do_bandpass 1 --cc_cutoff 0.18 --min_dist 300

Autopick output: 
/bin/bash: python: command not found

Convert command: 
emconvert --coordinates --from gautomatch --to xmipp --input Runs/000002_ProtImportMicrographs/micrographs.sqlite --output Tmp/2.outputMicrographs

Convert output: 
/bin/bash: emconvert: command not found

My LD_LIBRARY_PATH is empty by default and I dont change it in bashrc

pconesa commented 4 years ago

Thanks, and could you please verify that when changing the xmippEnv.json (removing /home/azazello/soft/miniconda3/envs/.scipion3env/lib from LD_LIBRARY_PATH) you are getting:

LD_LIBRARY_PATH: /usr/lib/x86_64-linux-gnu/stubs:/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/home/azazello/soft/scipion3/software/em/xmipp/lib

Without the conda lib path!

azazellochg commented 4 years ago

yes, I have { "LD_LIBRARY_PATH": "/usr/lib/x86_64-linux-gnu/stubs:/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu", "PATH": "/usr/lib/jvm/java-11-openjdk-amd64/bin", "CUDA": "True" }

azazellochg commented 4 years ago 
azazello@debian:~/soft/scipion3/software/em/xmipp$ grep -rnI "envs" *
bindings/python/xmipp_base.py:151:    from xmipp_conda_envs import XMIPP_CONDA_ENVS
bindings/python/xmipp_base.py:314:        """ expected kwargs:  see xmipp_conda_envs.py
build/xmipp.fish:6:set -px LD_LIBRARY_PATH /home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/lib /home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/bindings/python  /home/azazello/soft/miniconda3/envs/.scipion3env/lib
build/bindings/python/xmipp_base.py:151:    from xmipp_conda_envs import XMIPP_CONDA_ENVS
build/bindings/python/xmipp_base.py:314:        """ expected kwargs:  see xmipp_conda_envs.py
build/xmipp.bashrc:6:export LD_LIBRARY_PATH=/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/lib:/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/bindings/python::/home/azazello/soft/miniconda3/envs/.scipion3env/lib:$LD_LIBRARY_PATH
xmipp.bashrc:6:export LD_LIBRARY_PATH=/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/lib:/home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/bindings/python::/home/azazello/soft/miniconda3/envs/.scipion3env/lib:$LD_LIBRARY_PATH
xmipp.fish:6:set -px LD_LIBRARY_PATH /home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/lib /home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/bindings/python  /home/azazello/soft/miniconda3/envs/.scipion3env/lib
pconesa commented 4 years ago

Thanks do you still have the print line:

print("LD_LIBRARY_PATH: %s" % environ["LD_LIBRARY_PATH"])

If you click, with that xmipp environment file (without the conda lib), do you still get the same error....wait a second!,.

This line is actually doing that:

https://github.com/I2PC/scipion-em-xmipp/blob/release-20.07/xmipp3/__init__.py#L80

Comment: environ.addLibrary(Config.getPythonLibFolder(),position=pos)

Can you comment it and see.

azazellochg commented 4 years ago

Commented that one. Now I got this:

LD_LIBRARY_PATH: /usr/lib/x86_64-linux-gnu/stubs:/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/home/azazello/soft/scipion3/software/em/xmipp/lib
java -Xmx2g -Djava.library.path=/home/azazello/soft/scipion3/software/em/xmipp/lib -Dplugins.dir=/home/azazello/soft/scipion3/software/em/xmipp/bindings/java/imagej/plugins -cp lib/*:/home/azazello/soft/scipion3/software/em/xmipp/bindings/java/imagej/*:/home/azazello/soft/scipion3/software/em/xmipp/bindings/java/lib/* xmipp.viewer.particlepicker.training.SupervisedPickerRunner --input Runs/000002_ProtImportMicrographs/micrographs.sqlite --output Tmp/2.outputMicrographs --mode Automatic --classifier Tmp/2.outputMicrographs/picker.conf
Exception in thread "AWT-EventQueue-0" java.lang.UnsatisfiedLinkError: /home/azazello/soft/scipion3/software/em/xmippSrc-v3.20.07b1/build/lib/libXmippJNI.so: libhdf5.so.103: cannot open shared object file: No such file or directory
        at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
        at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2442)
        at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2498)
        at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694)
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2659)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:830)
        at java.base/java.lang.System.loadLibrary(System.java:1870)
        at xmipp.jni.MetaData.<clinit>(MetaData.java:98)
        at xmipp.viewer.particlepicker.training.model.SupervisedParticlePicker.loadEmptyMicrographsFromSqlite(SupervisedParticlePicker.java:446)
        at xmipp.viewer.particlepicker.training.model.SupervisedParticlePicker.loadEmptyMicrographs(SupervisedParticlePicker.java:371)
        at xmipp.viewer.particlepicker.ParticlePicker.<init>(ParticlePicker.java:160)
        at xmipp.viewer.particlepicker.training.model.SupervisedParticlePicker.<init>(SupervisedParticlePicker.java:76)
        at xmipp.viewer.particlepicker.training.model.SupervisedParticlePicker.<init>(SupervisedParticlePicker.java:69)
        at xmipp.viewer.particlepicker.training.model.SupervisedParticlePicker.<init>(SupervisedParticlePicker.java:122)
        at xmipp.viewer.particlepicker.training.SupervisedPickerRunner.run(SupervisedPickerRunner.java:28)
        at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:313)
        at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:770)
        at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
        at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:715)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
        at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:740)
        at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
        at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
        at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
        at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
        at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
        at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)
Exception in Tkinter callback
Traceback (most recent call last):
  File "/home/azazello/soft/miniconda3/envs/.scipion3env/lib/python3.8/tkinter/__init__.py", line 1883, in __call__
    return self.func(*args)
  File "/home/azazello/soft/scipion3/scipion-pyworkflow/pyworkflow/gui/form.py", line 1063, in _showWizard
    wizard.show(self.window)
  File "/home/azazello/soft/scipion-em-plugins/scipion-em-gautomatch/gautomatch/wizards.py", line 222, in show
    if myprops['applyChanges'] == 'true':
KeyError: 'applyChanges'
pconesa commented 4 years ago

I see, we need it for hdf5 and more...but I think at the same time is corrutping your bash. .

pconesa commented 4 years ago

Which OS is this? debian what? Your default shell is bash?

azazellochg commented 4 years ago

azazello@debian:~/soft/scipion3/software/em/xmipp$ cat /etc/debian_version bullseye/sid azazello@debian:~/soft/scipion3/software/em/xmipp$ uname -a Linux debian 5.7.0-1-amd64 #1 SMP Debian 5.7.6-1 (2020-06-24) x86_64 GNU/Linux

I don't use other shells

pconesa commented 4 years ago

@azazellochg , you are editing xmipp plugin in site-packages......does it mean is the pypi one?.

Probably there are differences between the released one and the one at release-20-07.

Are able to use the code from the branch?

azazellochg commented 4 years ago

Let me try a different PC tomorrow.

Best regards, Grigory

Grigory Sharov, Ph.D.

MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge CB2 0QH, UK. tel. +44 (0) 1223 267228 <+44%201223%20267228> e-mail: gsharov@mrc-lmb.cam.ac.uk

On Tue, Aug 25, 2020 at 3:11 PM Pablo Conesa notifications@github.com wrote:

@azazellochg https://github.com/azazellochg , you are editing xmipp plugin in site-packages......does it mean is the pypi one?.

Probably there are differences between the released one and the one at release-20-07.

Are able to use the code from the branch?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scipion-em/scipion-pyworkflow/issues/129#issuecomment-680049833, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVBPJQZFK7KNOKHDIHLGLLSCPBCBANCNFSM4QA2Y2JQ .

pconesa commented 4 years ago

For me, this fixes my case, so adding the position to the end should not break bash.

I think I got it. Could you please make a quick test:

At https://github.com/I2PC/scipion-em-xmipp/blob/release-20.07/xmipp3/__init__.py#L80

replace it by

environ.addLibrary(Config.getPythonLibFolder(),position=pos)

You'll also need the devel version fo the plugin (released one is not recent?)

pconesa commented 4 years ago

Hi, @azazellochg , did you manage to try this a different PC.

I've got a local change that might make sense, but if if does not fix anything I rather not push it.

azazellochg commented 4 years ago

Trying xmipp 20.7b1 on a different pc with or without position=pos didn't work:

java -Xmx2g -d64 -Djava.library.path=/lmb/home/gsharov/soft/scipion3/software/em/xmipp/lib -Dplugins.dir=/lmb/home/gsharov/soft/scipion3/software/em/xmipp/bindings/java/imagej/plugins -cp lib/*:/lmb/home/gsharov/soft/scipion3/software/em/xmipp/bindings/java/imagej/*:/lmb/home/gsharov/soft/scipion3/software/em/xmipp/bindings/java/lib/* xmipp.viewer.particlepicker.training.SupervisedPickerRunner --input Runs/000002_ProtImportMicrographs/micrographs.sqlite --output Tmp/2.outputMicrographs --mode Automatic --classifier Tmp/2.outputMicrographs/picker.conf
Preprocess command: NULL

Autopick command: 
python /lmb/home/gsharov/soft/scipion-em-plugins/scipion-em-gautomatch/gautomatch/run_gautomatch.py Runs/000002_ProtImportMicrographs/extra/KLH_Dataset_I_Test_0001.mrc Tmp/2.outputMicrographs/references.mrcs Tmp/2.outputMicrographs  --apixM 4.40 --ang_step 5 --diameter 500 --lp 30 --hp 1000 --gid 0 --apixT 4.40 --speed 2 --boxsize 150 --lsigma_cutoff 2.00 --lsigma_D 100 --lave_max 1.00 --lave_min -1.00 --lave_D 100 --detect_ice 1 --T_norm_type 1 --do_bandpass 1 --cc_cutoff 0.18 --min_dist 300

java.io.IOException: Cannot run program "bash": error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at java.lang.Runtime.exec(Runtime.java:621)
        at java.lang.Runtime.exec(Runtime.java:486)
        at xmipp.utils.XmippWindowUtil.executeCommand(XmippWindowUtil.java:298)
        at xmipp.viewer.particlepicker.training.model.GenericClassifier.execute(GenericClassifier.java:75)
        at xmipp.viewer.particlepicker.training.model.GenericClassifier.autopick(GenericClassifier.java:111)
        at xmipp.viewer.particlepicker.training.AutopickRunnable.run(AutopickRunnable.java:59)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 7 more
azazellochg commented 4 years ago

Running manually 

scipion run python /lmb/home/gsharov/soft/scipion-em-plugins/scipion-em-gautomatch/gautomatch/run_gautomatch.py Runs/000002_ProtImportMicrographs/extra/KLH_Dataset_I_Test_0001.mrc Tmp/2.outputMicrographs/references.mrcs Tmp/2.outputMicrographs  --apixM 4.40 --ang_step 5 --diameter 500 --lp 30 --hp 1000 --gid 0 --apixT 4.40 --speed 2 --boxsize 150 --lsigma_cutoff 2.00 --lsigma_D 100 --lave_max 1.00 --lave_min -1.00 --lave_D 100 --detect_ice 1 --T_norm_type 1 --do_bandpass 1 --cc_cutoff 0.18 --min_dist 300

works

pconesa commented 4 years ago

It's related to the available libraries (env) when running that command. Need to reproduce your case locally. Thanks, I'll come back, I hope with a solution.

pconesa commented 4 years ago

@azazellochg what gives you:

ldd /bin/bash

azazellochg commented 4 years ago

I'm back to my pc, so the original issues is : /bin/bash: python: command not found

@pconesa I don't think this has to do anything with bash:

(.scipion3env) azazello@debian:~/soft/scipion3/software/em/xmippSrc-3.20.07b1$ ldd /bin/bash
        linux-vdso.so.1 (0x00007fff4bb86000)
        libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f3a95c8c000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3a95c86000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3a95ac1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3a95e04000)

I've tried to run /bin/bash -c $PATH inside java and I got: /bin/bash: /home/azazello/soft/miniconda3/envs/.scipion3env/bin/python:/usr/lib/jvm/java-11-openjdk-amd64/bin:/home/azazello/soft/scipion3/software/em/xmipp/bin:

So I'm suspicious why /home/azazello/soft/miniconda3/envs/.scipion3env/bin/python is in PATH instead of /home/azazello/soft/miniconda3/envs/.scipion3env/bin/

scipion run env | grep PATH returns PATH=/home/azazello/soft/miniconda3/envs/.scipion3env/bin:/home/azazello/soft/miniconda3/condabin:/home/azazello/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

so it must be set by xmipp/java somewhere?

pconesa commented 4 years ago

Thanks @azazellochg , I understood wrongly the problem then. Certainly Xmipp is setting the environment before launching java and should be ../bin and not ../bin/python

azazellochg commented 4 years ago

I've printed PATH from xmipp3/__init__.py and there's no scipion3env at all, so it's set somewhere else.

pconesa commented 4 years ago

Yep,here https://github.com/I2PC/scipion-em-xmipp/blob/release-20.07/xmipp3/__init__.py#L57 nope.

Let me have a look at java code.

pconesa commented 4 years ago

In my machine the activation of the environment (no matter if virtualenv or conda is used) adds env/bin to the PATH, which make sense to me.

So from the beginning, it should be there: scipion3 run printenv | grep .scipion3env/bin

The problem could be here? https://github.com/I2PC/scipion-em-xmipp/blob/release-20.07/xmipp3/__init__.py#L62

Without a position....the variable is overwritten.

        environ.update({
            'PATH': pwem.Config.CUDA_BIN,
            'LD_LIBRARY_PATH': pwem.Config.CUDA_LIB
        }, position=pwutils.Environ.END)

Could you print the PATH variable, before and after these lines?

I've done it and previous PATH content is kept.

azazellochg commented 4 years ago 
azazello@debian:~$ scipion run printenv | grep .scipion3env/bin
PATH=/home/azazello/soft/miniconda3/envs/.scipion3env/bin:/home/azazello/soft/miniconda3/condabin:/home/azazello/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
PATH before update: /home/azazello/soft/miniconda3/envs/.scipion3env/bin:/home/azazello/soft/miniconda3/condabin:/home/azazello/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
PATH after update: /home/azazello/soft/miniconda3/envs/.scipion3env/bin:/home/azazello/soft/miniconda3/condabin:/home/azazello/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/cuda/bin

It looks like the PATH is mangled in java not in xmipp..

azazellochg commented 4 years ago

Let me check each update func in init

azazellochg commented 4 years ago

The following code in xmipp3 init:

    @classmethod
    def getEnviron(cls, xmippFirst=True):
        """ Create the needed environment for Xmipp programs. """
        environ = pwutils.Environ(os.environ)
        pos = pwutils.Environ.BEGIN if xmippFirst else pwutils.Environ.END

        environ.update({
            'PATH': pwem.Config.CUDA_BIN,
            'LD_LIBRARY_PATH': pwem.Config.CUDA_LIB
        }, position=pwutils.Environ.END)

        print("PATH before update: %s" % environ["PATH"])
        if os.path.isfile(getXmippPath('xmippEnv.json')):
            with open(getXmippPath('xmippEnv.json'), 'r') as f:
                compilationEnv = json.load(f)
            environ.update(compilationEnv)
        print("PATH after update: %s" % environ["PATH"])

        environ.update({
            'PATH': getXmippPath('bin'),
            'LD_LIBRARY_PATH': getXmippPath('lib'),
            'PYTHONPATH': getXmippPath('pylib')
                             }, position=pos)
        environ['XMIPP_HOME'] = getXmippPath()

        print("PATH after update2: %s" % environ["PATH"])
        # Add path to python lib folder
        #environ.addLibrary(Config.getPythonLibFolder(), position=pos)
        environ.addLibrary(Config.getPythonLibFolder())
        #print("LD_LIBRARY_PATH: %s" % environ["LD_LIBRARY_PATH"])
        #print("PATH: %s" % environ["PATH"])

        print("PATH after update3: %s" % environ["PATH"])
        return environ

returns:

PATH before update: /home/azazello/soft/miniconda3/envs/.scipion3env/bin:/home/azazello/soft/miniconda3/condabin:/home/azazello/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/cuda/bin
PATH after update: /usr/lib/jvm/java-11-openjdk-amd64/bin
PATH after update2: /usr/lib/jvm/java-11-openjdk-amd64/bin:/home/azazello/soft/scipion3/software/em/xmipp/bin
PATH after update3: /usr/lib/jvm/java-11-openjdk-amd64/bin:/home/azazello/soft/scipion3/software/em/xmipp/bin

So, loading xmippEnv.json indeed replaces the PATH but I dont see python there..

pconesa commented 4 years ago

Oh, no....you are not updated.

This was fixed some time ago by David maluenda, here: https://github.com/I2PC/scipion-em-xmipp/commit/8a31c7cc43a13177d3cc0f5887f354142fb2a046

and the current code has the pos parameter.:

https://github.com/I2PC/scipion-em-xmipp/blob/release-20.07/xmipp3/__init__.py#L67

Has the position specified...I don't see that bit in your code.

Am I right?

azazellochg commented 4 years ago

O my, I cannot believe it picks particles!!!!!!!!! Thank you @pconesa ! I've installed xmipp 2 weeks ago, but beta1 didnt have this change:

lrwxrwxrwx  1 azazello azazello        25 Aug  8 23:09 xmipp -> xmippSrc-v3.20.07b1/build
lrwxrwxrwx  1 azazello azazello        19 Aug  8 23:18 xmippSrc-3.20.07b1 -> xmippSrc-v3.20.07b1
drwxr-xr-x  5 azazello azazello      4096 Aug 16 14:42 xmippSrc-v3.20.07b1
-rw-r--r--  1 azazello azazello  19991020 Jul  8 17:20 xmippSrc-v3.20.07b1.tgz
pconesa commented 4 years ago

jejje, sorry I was focussed on a different issue, you got me into the right track. Thanks!

At least, it is fixed and is not something weird I was fearing.

pconesa commented 4 years ago

I think xmipp hasn't made any release recently. Probably is about time to do so.