GLIBCXX Not Found Error

[bdorney]

Trying to analyze latency data on cosmicstandtif.cern.ch

Package was already installed on cosmicstandtif.

Following compilation instructions found in Section 2.2.2 in the "GEM_Light_DQM_Reference_Guide.pdf"

Then launched the DAEMON following instructions in section 2.4.1 of the same manual.

Actual command was:

[GE11COSMICS@cosmicstandtif ~]$ python2.7 manage.py runserver cosmicstandtif.cern.ch:5017 >& log-lqdm.log &

Attaching log file showing errors. log-ldqm.txt

Also tried to run manually:

[GE11COSMICS@cosmicstandtif LightDQM]$ /home/GE11COSMICS/Software/DAQ/gem-light-dqm/gemtreewriter/bin/linux/x86_64_slc6/unpacker /tmp/run000180_LatencyScan_TIF_2016-11-11_chunk_768.dat sdram /home/GE11COSMICS/Software/DAQ/gem-light-dqm/gemtreewriter/bin/linux/x86_64_slc6/unpacker: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /home/GE11COSMICS/Software/DAQ/gem-light-dqm/gemtreewriter/bin/linux/x86_64_slc6/unpacker)

[mexanick]

Hi Brian,

this is not a code issue, this is an issue of your env variables. Please, refer to Jared and provide following: what is the version of light-dqm installed package? Is it release-v2? or release-v3? what was was sourced for env variables, including which version of ROOT you're running and what's in your PATH and LD_LIBRARY_PATH variables.

[jsturdy]

I suspect that the missing piece is to: source gem-light-dqm/settings.sh as this will add the updated gcc to LD_LIBRARY_PATH

To get the ldqm-browser up and running, the steps are:

source $BUILD_HOME/cmsgemos/setup/paths.sh
source $BUILD_HOME/gem-light-dqm/settings.sh
source $BUILD_HOME/ldqm-browser/setup.sh # fixed typo

then starting the daemon

python2.7 manage.py runserver blah

If these steps are not clear in the current documentation, we should make it more obvious.

[bdorney]

Okay so then the problem appears to be that the manual I have is out of date then. Under section 2.4.1 in the version I have it reads:

"RUNNING LIGHT DQM DAEMON/BROWSER source $BUILD_HOME/cmsgemos/setup/paths.sh source $BUILD_HOME/ldqm-browser/setup.sh cd $BUILD_HOME/ldqm-browser/LightDQM python2.7 manage.py runserver : >& logname.log &"

So sourcing the settings.sh files mentioned by jsturdy is not included in the documentation. Maybe there is an updated version of "GEM_Light_DQM_Reference_Guide.pdf" that could either be included in the repo (as either pdf, txt, or tex) and updated as changes are made (or uploaded here so I could reference it in the interim?)?

Also source $BUILD_HOME/ldqm-browser/settings.sh does not exist:

[GE11COSMICS@cosmicstandtif ldqm-browser]$ ls LightDQM README.md setup.sh

So I think my version of ldqm-browser is not up-to-date:

[GE11COSMICS@cosmicstandtif ldqm-browser]$ git branch -a

• develop master remotes/origin/HEAD -> origin/master remotes/origin/ajax-develop remotes/origin/develop remotes/origin/develop-osx remotes/origin/master

Should I do a pull request on the remote master or is one of the other branches what I should be using?

Thanks for the advice!

[jsturdy]

Hopefully @mexanick can confirm, but there may be two steps that can be done almost independently (which I think is how the documentation is outlined)

• Setting up the DQM code (gem-light-dqm)
  • This would not necessarily have to be done from either the same shell, nor every time, as the ldqm-browser
  • Did you make sure that this code was up to date (branch release-v3) and compiled (in doing this, you would have sourced settings.sh)
  • I suspect that the gem-light-dqm was built without the LD_LIBRARY_PATH including the newer gcc (but then again, I might expect this to have failed in a noticable way)
• Setting up and running the ldqm-browser
  • Sourcing the scripts mentioned in your issue report

[bdorney]

The compilation instructions I have did not mention to source the settings.sh script for the DQM (actually "settings.sh" is not in the document anywhere). I will try again attempting the above procedure. But if there is an updated version of the manual/documentation that I could follow it would definitely be useful.

I have [GE11COSMICS@cosmicstandtif gem-light-dqm]$ git branch -a ... ...

• release-v3 ... ...

It appears I am on the correct branch and up-to-date:

[GE11COSMICS@cosmicstandtif gem-light-dqm]$ git pull origin release-v3 From https://github.com/cms-gem-daq-project/gem-light-dqm

• branch release-v3 -> FETCH_HEAD Already up-to-date.

However if I look at the four branches for ldqm-browser available at either the central repo or mexanick's repo:

https://github.com/cms-gem-daq-project/ldqm-browser/ https://github.com/mexanick/ldqm-browser

There is no file:

source $BUILD_HOME/ldqm-browser/settings.sh

Maybe it was not committed when pushing to the remote repo?

[jsturdy]

Fixed a typo in my previous comment (ldqm-browser/setup.sh not ldqm-browser/settings.sh) Tried to follow my steps and actually still have the same error. Don't know what's missing, but clear the documentation is not being clear about what it expects.

@mexanick, I seem to recall that we had this issue before, but I don't remember what we needed to modify to have this work, any recollection?

Actually, it seems things are quite messy and must be cleaned up... Main issues:

• DQM requires non SLC6 version of ROOT and gcc
• cosmicstandtif has a different non-standard version of ROOT installed, so sourcing $BUILD_HOME/gem-light-dqm/settings.sh in addition to the prescribed recipe is necessary
• Furthermore, $BUILD_HOME/gem-light-dqm/settings.sh needs to be modified so that the nonstandard stdlibc++ path is first in LD_LIBRARY_PATH, otherwise it's not being picked up

All of this together, I think we have some clear action items

• ACTIONS
  • Make documentation clear about which non-standard programs are required
  • Make build/run process more robust
  • FUTURE WORK: Check for locally installed versions of python, gcc and ROOT
    • If sufficient versions exist, do not modify the environment variables
    • CURRENT: If non-standard versions are required, ensure that they are placed in the correct place in the PATHs by the relevant settings.sh/setup.sh (I suppose this means prepending, rather than appending)
  • FUTURE WORK: If universality of the build process can't be achieved (which is in principle fine), provide some form of requirements/dependencies file
  • Will be able to add this to the RPM stage hopefully
  • FUTURE WORK: create machine installation/setup procedure to put non-SLC standard versions as defaults
  • Will be good to have a prescription for this so we can just update a puppet config.
  • Can probably chat with the sys-admins to get some help on this
  • But then the tools will depend on having our specific setup, which for the most part should be avoided, best to have our machines set up in a way we know will work, but have each of the tools define properly dependencies

[bdorney]

Yes still cannot produce any output files.

When executing the daemon:

[GE11COSMICS@cosmicstandtif DAQ]$ source /home/GE11COSMICS/Software/DAQ/cmsgemos/setup/paths.sh [GE11COSMICS@cosmicstandtif DAQ]$ source $BUILD_HOME/gem-light-dqm/settings.sh [GE11COSMICS@cosmicstandtif DAQ]$ source $BUILD_HOME/ldqm-browser/setup.sh [GE11COSMICS@cosmicstandtif DAQ]$ python2.7 manage.py runserver cosmicstandtif.cern.ch:5017 >& log_lqdm_20161116.log &

I see errors being propagated to the log file (log_lqdm_20161116.txt) and when navigating to:

http://cosmicstandtif.cern.ch:5017/main/LatencySca/run/000181/AMC13-1/AMC-8/GTX-1/VFAT-9/hist/latencyScan/

Clicking on any of the buttons (for example "FiredChannels") does not seem to do anything.

I have tried to run the LightDQM manually but it appears to crash when trying to take a .raw.root file to a .analyzed.root file. You can see the commands I executed in lightDQM_Manual_No_Output.txt (also attached).

lightDQM_Manual_No_Output.txt

log_lqdm_20161116.txt

[mexanick]

The issue is resolved. The problem was in the DB name change which wasn't propagated to the local code used by @bdorney . Closing issue.