Closed zonca closed 3 years ago
My only guess is that maybe some nodes either don't have libssl or CVMFS isn't working quite right (buffer not big enough maybe?). Neither seems overly likely.
We'll be moving to a newer version of the base CVMFS environment within the next week or two, which may help if there's a problem with the distribution. Again, seems unlikely
thanks @bloer, something that I don't understand, maybe related:
when I run in the notebook, dask-scheduler
works fine and finds libssl
in the /lib64
folder, but /lib64
is not in LD_LIBRARY_PATH
, how does it know to look there?
oh sorry it is explained in the ldconfig
man: https://man7.org/linux/man-pages/man8/ldconfig.8.html
it is one of the standard directories.
But why doesn't that work in the dask container? it is exactly the same docker container I am running Jupyter in, so /lib64
is definitely there.
However instead of calling the env.sh
script from CVMFS, I am just setting PATH
LD_LIBRARY_PATH
and PYTHONPATH
.
@bloer do you think I need anything else from env.sh
?
nevermind! I was using the wrong container.... ok, now I have different errors I'm investigating.
ok, I think it is now working!
dask
workers always use 03.06
, will look for a better waydef get_python_path():
import sys
return sys.path
client.run(get_python_path)
{'tls://10.233.64.172:35025': ['/home/jovyan/dask-worker-space/dask-worker-space/worker-c1v5u3xz',
'/cvmfs/cdms.opensciencegrid.org/releases/centos7/V03-06/bin',
'/cvmfs/cdms.opensciencegrid.org/releases/centos7/V03-06/lib/python3.6/site-packages',
'/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.00-885ca/x86_64-centos7-gcc8-opt/lib',
'/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib',
'/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages',
'/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python36.zip',
'/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python3.6',
'/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python3.6/lib-dynload',
'/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages'],
'tls://10.233.65.62:43247': ['/home/jovyan/dask-worker-space/dask-worker-space/worker-dw8a74ch',
'/cvmfs/cdms.opensciencegrid.org/releases/centos7/V03-06/bin',
'/cvmfs/cdms.opensciencegrid.org/releases/centos7/V03-06/lib/python3.6/site-packages',
'/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.00-885ca/x86_64-centos7-gcc8-opt/lib',
'/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib',
'/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages',
'/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python36.zip',
'/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python3.6',
'/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python3.6/lib-dynload',
'/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages']}
Also the dashboard works fine:
@ziqinghong can you please test it and check if it works for you?
You should not have any package installed in ~/.local/lib
You can execute this test notebook, just replace js-XXX-YYY
with supercdms
:
https://gist.github.com/zonca/355a7ec6b5bd3f84b1413a8c29fbc877
Please report here any error.
@pibion @ziqinghong, I just tested this again now, it works fine, but it would be nice if someone else can test it within the next 2 weeks, especially I am interested to know if how it is setup now fits your needs or not.
We can schedule a Zoom meeting dedicated to this if you think it is useful (after you gave it a try).
ok, added docs to the usual README, pointing to #51 for later testing
I have configured the dask worker and the dask scheduler to mount the CVMFS volumes, and then modify the environment variables to load
V03-06
, they need to be hardcoded, I think it is fine for now, I can update that once in a while.However, I get some dynamic library errors.
When trying to launch the scheduler, I get:
However, if I try that in Jupyter, that works fine, it correctly picks up
libssl
fromlib64
:I also added
/lib64
toLD_LIBRARY_PATH
but it still fails, @bloer, would you have any suggestions?