search

dmwm / CRAB2

CRAB2
2 stars 11 forks source link

Python/sqlite problem: "undefined symbol: PyUnicodeUCS2_AsUTF8String" when running crab -create #236

Closed ericvaandering closed 10 years ago

ericvaandering commented 10 years ago

Original Savannah ticket 37733 reported by None on Thu Jun 12 02:59:23 2008.

I am trying to run some code under CMSSW_1_7_5 with crab, but sth is going wrong with python:

  1. Prepare CMSSW workspace and compile it.

cd /afs/cern.ch/user/l/lgrzanka/scratch/lgrzanka/tmp/test_grid/CMSSW_1_7_5/src source /afs/cern.ch/cms/sw/cmsset_default.sh scramv1 b

  1. Init grid (from http://cmsdoc.cern.ch/cms/aprom/www/top/CMS_VO.html) :

source /afs/cern.ch/cms/LCG/LCG-2/UI/cms_ui_env.sh

Check if certificate is valid (from https://edms.cern.ch/file/722398/1.2/gLite-3-UserGuide.html): grid-proxy-init -verify

Init proxy: voms-proxy-init -voms voce

  1. Init crab ( from https://twiki.cern.ch/twiki/bin/view/CMS/WorkBookRunningGrid )

source /afs/cern.ch/cms/ccs/wm/scripts/Crab/crab.sh

[lxplus215] /afs/cern.ch/user/l/lgrzanka/scratch/lgrzanka/tmp/test_grid/CMSSW_1_7_5/src > crab -create

crab: Istantiate DB Session : /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/py2-pysqlite/2.3.2/lib/python2.4/site-packages/pysqlite2/_sqlite.so: undefined symbol: PyUnicodeUCS2_AsUTF8String

and also:

[lxplus215] /afs/cern.ch/user/l/lgrzanka/scratch/lgrzanka/tmp/test_grid/CMSSW_1_7_5/src > python Python 2.3.4 (#1, Dec 11 2007, 21:18:01) [GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from pysqlite2 import test Traceback (most recent call last): File "<stdin>", line 1, in ? File "/afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/py2-pysqlite/2.3.2/lib/python2.4/site-packages/pysqlite2/test/init.py", line 25, in ? from pysqlite2.test import dbapi, types, userfunctions, factory, transactions,\ File "/afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/py2-pysqlite/2.3.2/lib/python2.4/site-packages/pysqlite2/test/dbapi.py", line 26, in ? import pysqlite2.dbapi2 as sqlite File "/afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/py2-pysqlite/2.3.2/lib/python2.4/site-packages/pysqlite2/dbapi2.py", line 27, in ? from pysqlite2._sqlite import * ImportError: /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/py2-pysqlite/2.3.2/lib/python2.4/site-packages/pysqlite2/_sqlite.so: undefined symbol: PyUnicodeUCS2_AsUTF8String >>>

ericvaandering commented 10 years ago

Comment by spiga on Thu Jun 12 03:46:19 2008

Hi

did you run your cmsenv before setting the crab environment ?

[eval scramv1 runtime -csh]

if not could you try and let we know?

Thanks Daniele

ericvaandering commented 10 years ago

Comment by lgrzanka on Thu Jun 12 03:51:26 2008

Yes, I tried to run [eval scramv1 runtime -sh] just after building [scramv1 b] and result is the same.

[lxplus215] /afs/cern.ch/user/l/lgrzanka/scratch/lgrzanka/tmp/test_grid/CMSSW_1_7_5/src > echo $PYTHONPATH /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/py2-pysqlite/2.3.2/lib/python2.4/site-packages:/afs/cern.ch/project/gd/LCG-share/3.1.4-1/external/opt/fpconst/lib/python2.3/site-packages:/afs/cern.ch/project/gd/LCG-share/3.1.4-1/external/opt/SOAPpy/lib/python2.3/site-packages:/afs/cern.ch/project/gd/LCG-share/3.1.4-1/glite/lib/python:/afs/cern.ch/project/gd/LCG-share/3.1.4-1/lcg/lib/python:/afs/cern.ch/exp/totem/scratch/lgrzanka/tmp/test_grid/CMSSW_1_7_5/lib/slc4_ia32_gcc345:/afs/cern.ch/exp/totem/scratch/lgrzanka/tmp/test_grid/CMSSW_1_7_5/python:/afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/cmssw/CMSSW_1_7_5/lib/slc4_ia32_gcc345:/afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/cmssw/CMSSW_1_7_5/python:/afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/pool/POOL_2_6_0-CMS5/slc4_ia32_gcc345/bin:/afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/pool/POOL_2_6_0-CMS5/slc4_ia32_gcc345/lib:/afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/coral/CORAL_1_9_0-CMS5/slc4_ia32_gcc345/python:/afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/coral/CORAL_1_9_0-CMS5/slc4_ia32_gcc345/lib:/afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/seal/SEAL_1_9_2-CMS16/slc4_ia32_gcc345/lib:/afs/cern.ch/cms/sw/slc4_ia32_gcc345/lcg/root/5.14.00g-CMS11/lib:/afs/cern.ch/cms/sw/slc4_ia32_gcc345/external/elementtree/1.1-CMS9/share/lib/python2.4/site-packages:/afs/cern.ch/cms/LCG/LCG-2/UI/lib/python2.2/site-packages:/afs/cern.ch/project/gd/LCG-share/3.1.4-1/glite/lib:/afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/python:/afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/DBSAPI:/afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/DLSAPI:/afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/PsetCode:/afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/ProdAgentApi:/afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/ProdCommon:/afs/cern.ch/project/gd/LCG-share/3.1.4-1/glite/lib

ericvaandering commented 10 years ago

Comment by spiga on Thu Jun 12 04:24:29 2008

Starting from a fresh shell and doing the sources in the right order all works fine to me.

could you start from a fresh shell too?

Cheers Daniele

ericvaandering commented 10 years ago

Comment by lgrzanka on Thu Jun 12 04:52:29 2008

I am starting from fresh lxplus shell (no predefined env variables):

  1. Change shell to bash:

bash

(python -> from pysqlite2 import test -> No module named pysqlite2)

  1. Init CMSSW enviroment variables

cd /afs/cern.ch/user/l/lgrzanka/scratch/lgrzanka/tmp/test_grid/CMSSW_1_7_5/src source /afs/cern.ch/cms/sw/cmsset_default.sh scramv1 b eval scramv1 runtime -sh

(python -> from pysqlite2 import test -> No module named pysqlite2)

  1. Init grid variables:

source /afs/cern.ch/cms/LCG/LCG-2/UI/cms_ui_env.sh voms-proxy-init -voms voce

(python -> from pysqlite2 import test -> No module named pysqlite2)

  1. Init crab variables:

source /afs/cern.ch/cms/ccs/wm/scripts/Crab/crab.sh

(python -> from pysqlite2 import test -> undefined symbol: PyUnicodeUCS2_AsUTF8String)

It seems, that sourcing crab.sh is reason of python problems. Default python version from SLC4 is 2.3, and sqlite library seems to be for python 2.4: /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/py2-pysqlite/2.3.2/lib/python2.4/site-packages (from echo $PYTHONPATH).

Which python are you using ?

More variables: echo $LANG gives en_US.UTF-8 echo $PYTHONPATH gives:

[lxplus221] /afs/cern.ch/user/l/lgrzanka/scratch/lgrzanka/tmp/test_grid/CMSSW_1_7_5/src > echo $PYTHONPATH | sed -e "s/:/\n/g" /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/py2-pysqlite/2.3.2/lib/python2.4/site-packages /afs/cern.ch/project/gd/LCG-share/3.1.4-1/external/opt/fpconst/lib/python2.3/site-packages /afs/cern.ch/project/gd/LCG-share/3.1.4-1/external/opt/SOAPpy/lib/python2.3/site-packages /afs/cern.ch/project/gd/LCG-share/3.1.4-1/glite/lib/python /afs/cern.ch/project/gd/LCG-share/3.1.4-1/lcg/lib/python /afs/cern.ch/exp/totem/scratch/lgrzanka/tmp/test_grid/CMSSW_1_7_5/lib/slc4_ia32_gcc345 /afs/cern.ch/exp/totem/scratch/lgrzanka/tmp/test_grid/CMSSW_1_7_5/python /afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/cmssw/CMSSW_1_7_5/lib/slc4_ia32_gcc345 /afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/cmssw/CMSSW_1_7_5/python /afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/pool/POOL_2_6_0-CMS5/slc4_ia32_gcc345/bin /afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/pool/POOL_2_6_0-CMS5/slc4_ia32_gcc345/lib /afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/coral/CORAL_1_9_0-CMS5/slc4_ia32_gcc345/python /afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/coral/CORAL_1_9_0-CMS5/slc4_ia32_gcc345/lib /afs/cern.ch/cms/sw/slc4_ia32_gcc345/cms/seal/SEAL_1_9_2-CMS16/slc4_ia32_gcc345/lib /afs/cern.ch/cms/sw/slc4_ia32_gcc345/lcg/root/5.14.00g-CMS11/lib /afs/cern.ch/cms/sw/slc4_ia32_gcc345/external/elementtree/1.1-CMS9/share/lib/python2.4/site-packages /afs/cern.ch/cms/LCG/LCG-2/UI/lib/python2.2/site-packages /afs/cern.ch/project/gd/LCG-share/3.1.4-1/glite/lib /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/python /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/DBSAPI /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/DLSAPI /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/PsetCode /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/ProdAgentApi /afs/cern.ch/cms/ccs/wm/scripts/Crab/CRAB_2_2_1/ProdCommon /afs/cern.ch/project/gd/LCG-share/3.1.4-1/glite/lib

BTW. Could you erase my comment #2 ?

ericvaandering commented 10 years ago

Comment by spiga on Thu Jun 12 05:35:44 2008

From a fresh shell the correct order as usual is the following:

1 source /afs/cern.ch/cms/LCG/LCG-2/UI/cms_ui_env.sh 2 eval scramv1 runtime -sh 3 source /afs/cern.ch/cms/ccs/wm/scripts/Crab/crab.sh

ericvaandering commented 10 years ago

Comment by lgrzanka on Thu Jun 12 07:36:24 2008

Thank for good advice ! This helped a lot :) Now I was able to run [crab -create], and [crab -submit].

Maybe it would be nice, to indicate in documentation, that the order of doing this steps is so important ?

ericvaandering commented 10 years ago

Comment by spiga on Thu Jun 12 07:58:16 2008

Dear Leszek

thanks for the suggestion.. but I don't know which doc. you are using.. since this info has been added years ago in the crab doc...

you can check all the official place: 1) https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideCrabHowTo 2) https://twiki.cern.ch/twiki/bin/view/CMS/WorkBook 3) http://cmsdoc.cern.ch/cms/ccs/wm/www/Crab/

cheers Daniele

please let me know if I can close the ticket.

ericvaandering commented 10 years ago

Comment by lgrzanka on Thu Jun 12 08:05:56 2008

I was using documentation from TWiki page https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideCrabHowTo

All this steps are described in correct way in section [Use CRAB installed at CERN], but it is not mentioned that order is important.

Most of the time I was working on shells, in which I did sourced CMSSW variables at the beggining (source /afs/cern.ch/cms/sw/cmsset_default.sh, scramv1 b, eval scramv1 runtime -sh). Sourcing grid and crab files is rare thing for me - this is the reason of changing order.

I think that it is worth to mention it in bold in documentation - especially for unexperienced user.

You can close the ticket. Thank you very much for quick response !!!

Cheers, Leszek

ericvaandering commented 10 years ago

Closed by spiga on Thu Jun 12 08:17:48 2008

ericvaandering commented 10 years ago

Comment by spiga on Thu Jun 12 08:17:48 2008

Added related note on https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideCrabHowTo

cheers Daniele