search

dmwm / DBS

CMS Dataset Bookkeeping Service
Apache License 2.0
7 stars 21 forks source link

Update pycurl version #602

Closed goughes closed 5 years ago

goughes commented 5 years ago

We found that the pycurl version for WMAgent and the version for DBS were different. This PR updates the DBS version to the correct one, keeping it consistent with WMAgent requirements file and the py2-pycurl RPM.

cmsdmwmbot commented 5 years ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-DBS-PR-test/108/artifact/artifacts/PullRequestReport.html

yuyiguo commented 5 years ago

@goughes Please tested DBS with the newer version.

goughes commented 5 years ago

Tested following the instructions here: https://github.com/dmwm/WMCore/wiki/DMWM-Packaging-with-PyPi#installing-dbs-client-and-pycurl-client-via-pypi-on-lxplus7

I uploaded the new version to PyPI with the version number 3.7.8.post1 so I could test.

[egough@lxplus705 ~]$ export SCRAM_ARCH=slc7_amd64_gcc630
[egough@lxplus705 ~]$ cd CMSSW_9_4_12/src
[egough@lxplus705 src]$ cmsenv
[egough@lxplus705 src]$ pip --no-cache-dir install --user dbs-client==3.7.8.post1
Collecting dbs-client==3.7.8.post1
<snip>
Successfully installed dbs-client-3.7.8.post1 pycurl-7.19.3 pycurl-client-3.7.8.post1

[egough@lxplus705 ~]$ python dbs-test.py 
Password for /afs/cern.ch/user/e/egough/.globus/userkey.pem: 
[{'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa1E31-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa8E29-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_7TeV_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHChcal-v1/GEN-SIM'}]

Let me know if you need anything else!

yuyiguo commented 5 years ago

I am quite busy with a DBS crisis and other stuff. I have only 25% FTE on CMS so it will take me a while to look into details of this.

From: Erik Gough notifications@github.com Reply-To: dmwm/DBS reply@reply.github.com Date: Friday, April 26, 2019 at 8:49 AM To: dmwm/DBS DBS@noreply.github.com Cc: Yuyi Guo yuyi@fnal.gov, Comment comment@noreply.github.com Subject: Re: [dmwm/DBS] Update pycurl version (#602)

Tested following the instructions here: https://github.com/dmwm/WMCore/wiki/DMWM-Packaging-with-PyPi#installing-dbs-client-and-pycurl-client-via-pypi-on-lxplus7https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_WMCore_wiki_DMWM-2DPackaging-2Dwith-2DPyPi-23installing-2Ddbs-2Dclient-2Dand-2Dpycurl-2Dclient-2Dvia-2Dpypi-2Don-2Dlxplus7&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=JD4cdtINSZd2Do5yr8dgUtiPIlYN2a6WLu_34rl59X0&s=V3tAXpxYYV3zSQjP5-hQ_H74zmBaPb78-ePtSPIaDuY&e=

I uploaded the new version to PyPI with the version number 3.7.8.post1 so I could test.

[egough@lxplus705 ~]$ export SCRAM_ARCH=slc7_amd64_gcc630

[egough@lxplus705 ~]$ cd CMSSW_9_4_12/src

[egough@lxplus705 src]$ cmsenv

[egough@lxplus705 src]$ pip --no-cache-dir install --user dbs-client==3.7.8.post1

Collecting dbs-client==3.7.8.post1

Successfully installed dbs-client-3.7.8.post1 pycurl-7.19.3 pycurl-client-3.7.8.post1 [egough@lxplus705 ~]$ python dbs-test.py Password for /afs/cern.ch/user/e/egough/.globus/userkey.pem: [{'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa1E31-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa8E29-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_7TeV_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHChcal-v1/GEN-SIM'}] Let me know if you need anything else! — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
yuyiguo commented 5 years ago

@goughes Erik, Looked at this PR, I have a few questions: 1."Successfully installed dbs-client-3.7.8.post1 pycurl-7.19.3 pycurl-client-3.7.8.post1" The instruction said that the special pycurl library was brought in by CMSSW. But we install a generic pycurl anyway?

  1. The test you did was only to proved that the connection worked. Since DBS is a complicated system, we need to test the speed of the connection and how this change will affect the entire DBS operation.

I got a lot of things going on DBS land , this pycurl change may sit on my list for a while. It would be too much for you to run a full DBS evaluation. So if you and Alan (@amaltaro ) want , I could commit this PR for you know, but this change will only for pip. WMAgent is going to run a different versions of DBS for a while before I can apply the newer version of pycurl to the rest.

Yuyi

amaltaro commented 5 years ago

Hi Yuyi,

at least one of the DBS services rely on the COMP py2-pycurl spec: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/py2-pycurl.spec

which is currently using version 7.19.3. So this PR is actually just fixing a typo made in Erik's previous PR (already merged). Sorry for the extra work, but it's completely safe to merge it, it won't change a bit of the DBS service, instead it will only provide a correct version for pycurl (which is at the moment only used when building a pypi release).

yuyiguo commented 5 years ago

@amaltaro OK, Alan. This was a typo correction instead of updating DBS to a new pycurl version. Just chcked that DBS already using 7.19.3. Committing it now.

yuyiguo commented 5 years ago

@goughes All done.