CMS: lost mount point to CVMFS in docker container

[diemort]

Dear Devs,

I've been running CMSSW_5_3_32/src via docker and it worked fine in its first run. However, after exiting the container and running it again, I see that I've lost the mount point to CVMFS and I cannot run any code base on the GTs or access the conditions DB.

Is there a way to re-mount CVMFS in this case?

I've tried to run the cvmfs script but none are installed in the container. Besides, autofs is also missing and cannot be ran.

Cheers,

--Gustavo

[diemort]

Just to clarify the issue:

cmsusr@a4512ded7f99 ~/CMSSW_5_3_32/src/MCDY2011 $ cmsRun gensimDY.py ----- Begin Fatal Exception 28-Nov-2019 12:58:20 CET----------------------- An exception of category 'StdException' occurred while [0] Constructing the EventProcessor [1] Constructing ESSource: class=PoolDBESSource label='GlobalTag' Exception Message: A std::exception was thrown. Connection on "sqlite_file:/cvmfs/cms-opendata-conddb.cern.ch/START53_LV6A1.db" cannot be established ( CORAL : "ConnectionPool::getSessionFromNewConnection" from "CORAL/Services/ConnectionService" ) ----- End Fatal Exception -------------------------------------------------

[katilp]

Thanks for reporting! Is this maybe connected to the ip/port whitelisting which you observed for the VM?

[diemort]

Hi Kati,

I just forwarded your message about ip/ports to the IT team of my Institution.

My approach with docker is a way to avoid my current issue with the ip/ports in the CERN VM. The docker container worked just fine yesterday when I installed CMSSW_5_3_32 in the first time:

docker run --name opendata -it cmsopendata/cmssw_5_3_32 /bin/bash

from the outer shell.

Yesterday, after analyzing a few events from the 2011 datasets, I exited the container as usual. Today I reconnect to the container, but I cannot get the GTs since the mounting point to CVMFS is not working.

It may be related to the ip/ports issue, but I wouldn't understand why it worked yesterday and not today.

I've tried to follow the instruction here to manually mount CVMFS in my docker, but they didn't work.

Is there a way to re-mount CVMFS from inside the container? Or it should be done from the host?

Cheers,

--Gustavo

[katilp]

The container we have does not yet have a proper cvmfs access, that's ongoing work. Did it really read something from GTs before, or did it just happen that it fails when it needs something from there only after some events? This explains if you get further in the dataset on you second try.

[diemort]

I'm following the instructions here, where we need to make a link to the GT stored in CVMFS. It worked once, but now I get the error message as shown in the comment above.

[clelange]

Hello @diemort - the cmsopendata/cmssw_5_3_32 container does not know anything about CVMFS, which means that you cannot remount it from within the container. I'm a bit confused how you actually manage to access the conditions database SQL file on CVMFS when using this container and the command you provided above.

If you mount CVMFS on the computer that has docker installed, you need to run the following command (assuming that you mount CVMFS on your computer at /cvmfs):

docker run --name opendata -v /cvmfs:/cvmfs -it cmsopendata/cmssw_5_3_32 /bin/bash

Should the /cvmfs/cms-opendata-conddb.cern.ch mount on your computer not be available anymore, you should probably just restart the cvmfs service/remount cvmfs.

[diemort]

Hello Clemens,

I've tried to run this container in the computer I have CVMFS mounted, but it does not work. Maybe there's some extra config to allow the container to access the CVMFS area.

Actually, my CVMFS mount is not showing cms-opendata-conddb.cern.ch:

> cvmfs_config probe Probing /cvmfs/cms.cern.ch... OK Probing /cvmfs/oasis.opensciencegrid.org... OK

I've tried to include the DB repo:

CVMFS_REPOSITORIES=cms-opendata-conddb.cern.ch,grid.cern.ch,cms.cern.ch

but it does not mount:

ll /cvmfs/ drwxr-xr-x. 63 cvmfs 4.0K Aug 29 2014 cms.cern.ch drwxr-xr-x. 4 cvmfs 4.0K Feb 26 2016 config-osg.opensciencegrid.org drwxr-xr-x. 21 cvmfs 4.0K Nov 16 2017 oasis.opensciencegrid.org

Am I missing something?

Cheers, Gustavo

PS: I've also tried your standalone container from clelange/cmssw-docker, but I got an error. Issue posted in the GitHub repo.

[caredg]

Hello, I was recently testing this and saw similar instabilities. I became convinced that the magic is to use CVMFS_HTTP_PROXY=DIRECT in the /etc/cvmfs/default.local file.

[clelange]

Hi @caredg - if outside CERN, you must use CVMFS_HTTP_PROXY=DIRECT, some of the other servers are not accessible from the outside. I'm a bit confused which image you're using. Is it the one with cvmfs in its name or the plain slc6-cms image? The image that starts CVMFS from within the container only does so when starting the container the first time, but not when you reconnect.

[caredg]

Hi @clelange. I see, with the servers. Sometimes it worked, and sometimes it didn't.... I was mentioning the CVMFS_HTTP_PROXY=DIRECT config line just in case it is useful (it worked for me, and now I know why). The image I use is cmsopendata/cmssw_5_3_32 (so it does not have cvmfs, that is why I mount it from my local host). It works fine now, btw.

[diemort]

Dear all, following this alternative of using CVMFS_HTTP_PROXY=DIRECT and the explanation about availability of the servers outside CERN, I wonder if it would help to use the conventional virtual machine (not a docker container).

I explain: I started using the docker container because the OpenData virtual machine could not connect to the CERN server when inside my Institution network. I believe this was some issue with my Institution firewall, since it work from home. The issue was related to not accessing cvmfs-stratum-one.cern.ch with IP 188...*. The IT team tested and recommended for me the docker option. At the moment I cannot test it again, but I wonder if CVMFS_HTTP_PROXY=DIRECT would overcome that issue as well.

[clelange]

I think we should disentangle two things here because I'm a bit confused now. There is the conditions database, which can be accessed directly (even though we do not document this), and then there are the conditions DB SQL file extracts, which are located in CVMFS mounted at /cvmfs/cms-opendata-conddb.cern.ch/. One could also simply download the SQL file and then not bother with CVMFS at all. This might be the easier solution, in particular when already downloading several Gigabytes for the full CMSSW container? What is it that you'd like to do?