search

dmwm / DBS

CMS Dataset Bookkeeping Service
Apache License 2.0
7 stars 21 forks source link

DBS pypi packaging #591

Closed goughes closed 5 years ago

goughes commented 5 years ago

This PR is related to #540. Do not merge yet as it is a work in progress. I have added requirements.txt and setup scripts that can be used to upload the DBS client and Pycurl client to pypi. These are currently available on pypi and can be installed via pip.

Some docs: https://github.com/dmwm/WMCore/wiki/DMWM-Packaging-with-PyPi PyPI page for DMWM: https://pypi.org/user/cms-oc-dmwm/

Regarding versioning, pypi packages need to follow PEP 440: https://www.python.org/dev/peps/pep-0440/ DBS tags do not follow this scheme, so we should discuss if/how they can be changed.

cmsdmwmbot commented 5 years ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-DBS-PR-test/85/artifact/artifacts/PullRequestReport.html

cmsdmwmbot commented 5 years ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-DBS-PR-test/86/artifact/artifacts/PullRequestReport.html

cmsdmwmbot commented 5 years ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-DBS-PR-test/90/artifact/artifacts/PullRequestReport.html

cmsdmwmbot commented 5 years ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-DBS-PR-test/91/artifact/artifacts/PullRequestReport.html

goughes commented 5 years ago

Spoke with Rucio devs regarding how they build packages and they pointed me to this: https://github.com/rucio/rucio/blob/master/tools/build_sdist_and_upload.sh I added a similar script for DBS.

That leaves this question. What other software from the DBS repo needs to be available in pypi? dbs-web? dbs-migration?

cmsdmwmbot commented 5 years ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-DBS-PR-test/93/artifact/artifacts/PullRequestReport.html

yuyiguo commented 5 years ago

Only DBS client and pycurl client.

amaltaro commented 5 years ago

@goughes Erik, I'm wondering whether it looks cleaner to simply add that setupfunctions code to each of those setup* scripts? Just so we keep all code related to building pypi packages in the same place.

Maybe we can "blindly" open that file, if requirements cannot be found an exception will be raised and the packaging process will be stopped (that will save us a few lines in the code :))

yuyiguo commented 5 years ago

@goughes Erik, Could you please add instructions on how to install DBS client via pypi ? DBS Client currently deployed with Crab. The instruction https://twiki.cern.ch/twiki/bin/viewauth/CMS/DBS3APIInstructions points how to get DBS Client through crab. It would be nice that you could add the instructions via pypi there. Thanks a lot.

goughes commented 5 years ago

@amaltaro You are right. I have removed setup_functions so all the code needed to build a package is in a the individual setup files.

cmsdmwmbot commented 5 years ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-DBS-PR-test/94/artifact/artifacts/PullRequestReport.html

cmsdmwmbot commented 5 years ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-DBS-PR-test/96/artifact/artifacts/PullRequestReport.html

goughes commented 5 years ago

@amaltaro @yuyiguo I updated the instructions here for how to build and upload new PyPI package and install DBS client via PyPI. Can you take a look? DM me on Slack if you want the password for uploading. https://github.com/dmwm/WMCore/wiki/DMWM-Packaging-with-PyPi#packaging-dbs-applications

cmsdmwmbot commented 5 years ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-DBS-PR-test/98/artifact/artifacts/PullRequestReport.html

yuyiguo commented 5 years ago

@goughes Erik, It seems this pypi installing DBS is really useful to end users. Currently DBS client is distributed with crab client which is not all the DBS users need. I tried to install DBS client on my laptop, but I did not have virtualenv . The end users' own machines usually may not have virtualenv. If you could add instruction on installing virtualenv and remove the dependency on CMS library, then we could advertise this.

amaltaro commented 5 years ago

@goughes @yuyiguo I added a couple of commands for installing pip and virtualenv, then after installing libcurl-devel I managed to install dbs-client. However, calling the dbsApis fail with the pycurl.error error you already reported. Trying the cvmfs tricky didn't help either.

However, if I instead use a pair of cert/key service certificate files, then it works just fine.

yuyiguo commented 5 years ago

Thanks Alan for updating the instruction. Could you please also add the instructions on how to use cert/key files to make it really works?

From: Alan Malta Rodrigues notifications@github.com Reply-To: dmwm/DBS reply@reply.github.com Date: Wednesday, February 6, 2019 at 7:35 AM To: dmwm/DBS DBS@noreply.github.com Cc: Yuyi Guo yuyi@fnal.gov, Mention mention@noreply.github.com Subject: Re: [dmwm/DBS] DBS pypi packaging (#591)

@gougheshttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_goughes&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=ehW9ZX3UfTe33RD6HPabmNZR_YUmYNPZ79bxdwtyu0I&s=8w9YtTfPqYIx151phQDBTcWJ4zDJTPYWA2SsTMkPS_4&e= @yuyiguohttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_yuyiguo&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=ehW9ZX3UfTe33RD6HPabmNZR_YUmYNPZ79bxdwtyu0I&s=zerB38-5O1PiUjH6PLYHxATAEjG7gkyi3M1Ioi0jBfM&e= I added a couple of commands for installing pip and virtualenv, then after installing libcurl-devel I managed to install dbs-client. However, calling the dbsApis fail with the pycurl.error error you already reported. Trying the cvmfs tricky didn't help either.

However, if I instead use a pair of cert/key service certificate files, then it works just fine.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D461024747&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=ehW9ZX3UfTe33RD6HPabmNZR_YUmYNPZ79bxdwtyu0I&s=Pj4JGpMK5SGt_BQ_t1m3dDPhKtcQyFiOfa8mUhXGwaQ&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABsXTniFPsrdXEeDcywKsUGGUkK7FCOXks5vKtofgaJpZM4Z8HPH&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=ehW9ZX3UfTe33RD6HPabmNZR_YUmYNPZ79bxdwtyu0I&s=LKW4244QBvGOcpADQJIbIiIxxCw65MIlRVtgyvoCc0I&e=.

amaltaro commented 5 years ago

For that you would need to have access to a service/robot certificate. So I don't know how we can make it work for a "mortal" user.

goughes commented 5 years ago

Seems like this might be related to some special sauce CMS puts into curl... anyone know who I could ask for some additional info?

(dbs7) [egough@lxplus031 ~]$ python dbs-test.py 
Traceback (most recent call last):
  File "dbs-test.py", line 3, in <module>
    ds = dbs.listDatasets(dataset = '/MinBias/Summer*/*')
  File "/afs/cern.ch/user/e/egough/dbs7/lib/python2.7/site-packages/dbs/apis/dbsClient.py", line 758, in listDatasets
    return self.__callServer("datasets", params=kwargs)
  File "/afs/cern.ch/user/e/egough/dbs7/lib/python2.7/site-packages/dbs/apis/dbsClient.py", line 197, in __callServer
    self.http_response = method_func(self.url, method, params, data, request_headers)
  File "/afs/cern.ch/user/e/egough/dbs7/lib/python2.7/site-packages/RestClient/RestApi.py", line 34, in get
    return http_request(self._curl)
  File "/afs/cern.ch/user/e/egough/dbs7/lib/python2.7/site-packages/RestClient/RequestHandling/HTTPRequest.py", line 57, in __call__
    curl_object.perform()
pycurl.error: (35, 'Peer does not recognize and trust the CA that issued your certificate.')
(dbs7) [egough@lxplus031 ~]$ export LD_LIBRARY_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/curl/7.59.0/lib
(dbs7) [egough@lxplus031 ~]$ python dbs-test.py 
[{'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa1E31-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa8E29-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_7TeV_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHChcal-v1/GEN-SIM'}]
amaltaro commented 5 years ago

@bbockelm or @belforte might know it better, maybe they can shed some lights on this issue.

belforte commented 5 years ago

I am not sure I understand. Is this : things fail with curl from /usr/bin but works with curl from CMSSW ? The libcurl (and curl) which come with SL6 and SL7 are known not to work with SSL. One needs the version from CMS. There may be other ways of course... but I do not know them. As long as we require SSL for accessing CMS web services... we need a libcurl built with SL suppport. E.g. my ubuntu laptop has a 'good' curl out of the box. SL6/7 alas not. That at least is what I have heard so far. Did I answer the question ?

belforte commented 5 years ago

example (I used one curl line from CRAB op/debug twiki). In all cases after voms-proxy-init -voms CMS and haveing defined X509_USER_PROXY

amaltaro commented 5 years ago

Thanks, Stefano. Yes, I think you did, but I let Erik discuss these details with you since he's the one really investigating this issue.

I do have a few comments though:

One needs the version from CMS.

which is available either in the CMSSW environment or in our CMS COMP one. Then looking at cmsdist/comp_gcc630, I see dbs3-client depends on dbs3-pycurl-client, which depends on curl, and the curl spec is built with SSL support. So I still fail to see why it doesn't work.

vkuznet commented 5 years ago

I think it is missing environment settings in pypi installation. In CMSSW or CMS COMP every package has etc/profile.d/init.sh and etc/profile.d/dependencies.sh files which sets PYTHONPATH and LD_LIBRARY_PATH properly. Since now you're installing via pypi these settings are missing in your setup. What you see is that by running python dbs-test.py the latter script will load ssl and take system curl shared library. As you demonstrated when you setup LD_LIBRARY_PATH to point to proper curl library the behavior is fixed. But unfortunately it is not proper way to do it due to recursive chain of dependencies which we set through etc/profile.d/init.sh. You need to source this script which will go through all packages and setup full PYTHONPATH and LD_LIBRARY_PATH. I doubt that there is a mechanism in pypi installation to load this script by default.

vkuznet commented 5 years ago

But I also think that CMSSW folks (David Lange, Shahzad) resolved somehow this issue since they should have identical problem installing and running CMSSW packages in pypi/virtualenv. I would suggest to talk to them about it.

belforte commented 5 years ago

Well.. what is exactly the goal here ? Indeed CMS_COMP provides the proper libcurl and if the goal is to use pip to run DBS Client in the CMS_COMP environment w/o esplictely sourcing the CMS_COMP setup first, you need to tackle what Valenting pointed out.

Are you looking into getting rid of CMS_COMP for WMA ? If not, just source it etc/profile.d/initl.sh and go.

I thought you wanted to make it possible to run DBS Client "on any SL6/7 box', in that case the first question is 'where to get libcurl'.

But if the goal is to make it easy to use for CMS users, we can assume that they have the CMSSW environment defined and IIUC simply use pip to add DBS client to it w/o bothering with a full CMS_COMP distribution (which would be the current proper way). As Yuyi pointed out, getting DBS Client python files extracted from gtihub and sticked inside CRAB Client CVMFS directories is not so nice, while I do expect a technical problem (just add a few python files to the ones for CRAB), for sure it makes version control difficult and is somehow un-natural since CRAB Client does not talk to DBS.

goughes commented 5 years ago

Thanks for the comments everyone.

The original goal of this PR was to get DBS client and pycurl client into pypi so they could be pulled as dependencies when installing WMAgent via pip. I think we have met that goal as the curl issue with user certs or proxies I describe above will not exist in that scenario.

Giving users to install a working dbs-client on any client would be quite tricky and is probably out of the scope of this PR. At a minimum, I can put some instructions for installing dbs-client via pip on lxplus7 with the CMSSW environment.

belforte commented 5 years ago

Tahnks @goughes When you have instructions for users install on top of CMSSW, it may be good to add to existing twiki: https://twiki.cern.ch/twiki/bin/view/CMS/DBS3APIInstructions

amaltaro commented 5 years ago

@goughes can you please follow the instructions you wrote down and upload the latest dbs-client to Pypi (3.7.8)? If everything goes well, then I'd suggest we merge this PR and follow up the curl/architecture limitations in another ticket. As Erik mentioned, the main goal was to get WMAgent deployable via Pypi.

@yuyiguo if you agree, then let's wait for Erik's confirmation and we can move on with this.

yuyiguo commented 5 years ago

Alan, That is fine with me. But what I understood was that DBS client was used as guinea pig to deploy DM code via Pypi. Cheers, Yuyi

From: Alan Malta Rodrigues notifications@github.com Reply-To: dmwm/DBS reply@reply.github.com Date: Wednesday, February 13, 2019 at 3:30 AM To: dmwm/DBS DBS@noreply.github.com Cc: Yuyi Guo yuyi@fnal.gov, Mention mention@noreply.github.com Subject: Re: [dmwm/DBS] DBS pypi packaging (#591)

@gougheshttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_goughes&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=NAjDDJKzf2mLz2XDh083dmHp9YPBmIDLEgcyG1YVsvI&s=1hfDmWaJMub3muA1zwKiaKlXMaKfJMJcstxVk4IFqtI&e= can you please follow the instructions you wrote down and upload the latest dbs-client to Pypi (3.7.8)? If everything goes well, then I'd suggest we merge this PR and follow up the curl/architecture limitations in another ticket. As Erik mentioned, the main goal was to get WMAgent deployable via Pypi.

@yuyiguohttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_yuyiguo&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=NAjDDJKzf2mLz2XDh083dmHp9YPBmIDLEgcyG1YVsvI&s=qUT-xMtb7EcRTfzi1Zbqo68Nob2KfZI-Th6Rfj3y8ms&e= if you agree, then let's wait for Erik's confirmation and we can move on with this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D463125042&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=NAjDDJKzf2mLz2XDh083dmHp9YPBmIDLEgcyG1YVsvI&s=aon7HLd8Mg42eWkq1_SgO4_ThLJGhgG4t1OLzphlp2o&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABsXTgcy2XOJRJZqUtsWu9s-2DkhQ3nopoks5vM9sZgaJpZM4Z8HPH&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=NAjDDJKzf2mLz2XDh083dmHp9YPBmIDLEgcyG1YVsvI&s=MqqbRtOaibWACO6uMpvQI2T0-LWNjKSsPbAQX1pWtsM&e=.

goughes commented 5 years ago

I uploaded the 3.7.8 version of dbs client and pycurl client to PyPi. I was also able to install and make a successful query to the DBS api using the updated instructions I put here https://github.com/dmwm/WMCore/wiki/DMWM-Packaging-with-PyPi#installing-dbs-client-and-pycurl-client-via-pypi-on-lxplus7

I did not have any success mixing a CMSSW env with a python virtualenv, so the instructions use the --user option of pip to install the package to the .local dir in the user's home directory.

belforte commented 5 years ago

isn't this what anybody w/o root access will get by default anyhow ?

I did not have any success mixing a CMSSW env with a python virtualenv, so the instructions use the --user option of pip to install the package to the .local dir in the user's home directory.

goughes commented 5 years ago

If you don't specify --user it tries to install into the system python directories and you get an error.

OSError: [Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/RestClient'

belforte commented 5 years ago

thanks @goughes and sorry for noise, I was using confused memories.

amaltaro commented 5 years ago

@yuyiguo would you have any description/documentation already available giving a summary or overview of these 2 packages? Such that Erik can update it on the Pypi page.

In addition to that, can we now merge this PR?

yuyiguo commented 5 years ago

@amaltaro I don't have anything on the pypi package. We depend on the author to write the package summary and document as well as unit tests, deployment tests if they are applicable. If everything is done, feel free to commit.

amaltaro commented 5 years ago

Well, the author of DBS packages/services is you ;-) Erik simply did the upload and the setup to get this available. Anyhow, I'm not asking you to give us anything now, this is for the future... Merging this one would be a good start IMO.

yuyiguo commented 5 years ago

@amaltaro I did not write the pypi part. Anyone contribute to any package in CMS is more than welcome as I understood. However, Contributing to a package is more than writeing the code, it includes document, and tests. I hope we are in the same page.

belforte commented 5 years ago

It may be off topic, but I find ti REALLY GREAT that now I can run dbs client on my Ubuntu laptop ! (latest Ubuntu has proper libcurl out of the box).

I am interested also in wmcore, I see that pip has 1.1.1rc7, do you plan to make and keep that up to date ?

vkuznet commented 5 years ago

Stefano, as I reported on meeting, we should be careful with installations until proper license will be in place. Best, Valentin.

On 0, Stefano Belforte notifications@github.com wrote:

It may be off topic, but I find ti REALLY GREAT that now I can run dbs client on my Ubuntu laptop ! (latest Ubuntu has proper libcurl out of the box).

I am interested also in wmcore, I see that pip has 1.1.1rc7, do you plan to make and keep that up to date ?

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DBS/pull/591#issuecomment-465201810

belforte commented 5 years ago

fine with being careful, i am simply a bit lost as to what the goal is and how this impacts CRAB

On 19/02/2019 10:32, Valentin Kuznetsov wrote:

Stefano, as I reported on meeting, we should be careful with installations until proper license will be in place. Best, Valentin.

On 0, Stefano Belforte notifications@github.com wrote:

It may be off topic, but I find ti REALLY GREAT that now I can run dbs client on my Ubuntu laptop ! (latest Ubuntu has proper libcurl out of the box).

I am interested also in wmcore, I see that pip has 1.1.1rc7, do you plan to make and keep that up to date ?

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DBS/pull/591#issuecomment-465201810

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmwm/DBS/pull/591#issuecomment-465207134, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwK2adbW4DxSie2OGaPcZ7QGiA0XUyRks5vPCcXgaJpZM4Z8HPH.

yuyiguo commented 5 years ago

The same question here. What is the goal of all this? What kind license do I need ? But it is still a good news that Stefano can run it out of box w/o CMS specific software decency. Can we improve DBS pypi package to get the newest libcurl? Thanks, Yuyi

From: Stefano Belforte notifications@github.com Reply-To: dmwm/DBS reply@reply.github.com Date: Tuesday, February 19, 2019 at 11:03 AM To: dmwm/DBS DBS@noreply.github.com Cc: Yuyi Guo yuyi@fnal.gov, Mention mention@noreply.github.com Subject: Re: [dmwm/DBS] DBS pypi packaging (#591)

fine with being careful, i am simply a bit lost as to what the goal is and how this impacts CRAB

On 19/02/2019 10:32, Valentin Kuznetsov wrote:

Stefano, as I reported on meeting, we should be careful with installations until proper license will be in place. Best, Valentin.

On 0, Stefano Belforte notifications@github.com wrote:

It may be off topic, but I find ti REALLY GREAT that now I can run dbs client on my Ubuntu laptop ! (latest Ubuntu has proper libcurl out of the box).

I am interested also in wmcore, I see that pip has 1.1.1rc7, do you plan to make and keep that up to date ?

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DBS/pull/591#issuecomment-465201810https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465201810&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=aYAUobOHc99ofOip-rRR49snFUGN1RDkOZ5EMOKv6no&e=

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmwm/DBS/pull/591#issuecomment-465207134https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465207134-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=2ooGSj0gpAw2Ptka9Ye-KV4atbuEyAeqgqK8fKNgK08&e=, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwK2adbW4DxSie2OGaPcZ7QGiA0XUyRks5vPCcXgaJpZM4Z8HPHhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABwK2adbW4DxSie2OGaPcZ7QGiA0XUyRks5vPCcXgaJpZM4Z8HPH-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=h_R4nqSKTgFUIN7dGZ7gETpqmPokBXtiPjSCOTD5f8w&e=.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465220822&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=vY7sHw9hIaeTJ0bumpGSbvbKIy75PwSszdSJ9y6LyiY&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABsXTiuS473IEbNxNzWEoBwZpQlt3Yrbks5vPC5YgaJpZM4Z8HPH&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=aJPX5wVS63cMoA3fPmI_dP7X9zw5l-XF6z2zEYA8X4A&e=.

vkuznet commented 5 years ago

Yuyi, on a WMA meeting we discussed this. Original goal was to simplify WMA deployment and not a public release of DBS client.

Regarding license, you need to understand that whatever you release to public domain (in this case pypi) can be used for different purposes. All CMS software is open source, but according to this note:

https://indico.cern.ch/event/791214/attachments/1783962/2906417/CBnewsletter-CB122-February-2019.pdf

the collaboration board discuss a possibility to release all CMS software under apache 2 license instead of GPL one (may be it is default one), see item 7.

I suggested on a meeting that our release of software to public should be adopted with certain license and I asked Alan to clarify what it should be.

As developers we should be aware of how our software is distributed to avoid possible issues. That's why I made my suggestions that some license should be added to pypi packages.

Best, Valentin.

On 0, Yuyi Guo notifications@github.com wrote:

The same question here. What is the goal of all this? What kind license do I need ? But it is still a good news that Stefano can run it out of box w/o CMS specific software decency. Can we improve DBS pypi package to get the newest libcurl? Thanks, Yuyi

From: Stefano Belforte notifications@github.com Reply-To: dmwm/DBS reply@reply.github.com Date: Tuesday, February 19, 2019 at 11:03 AM To: dmwm/DBS DBS@noreply.github.com Cc: Yuyi Guo yuyi@fnal.gov, Mention mention@noreply.github.com Subject: Re: [dmwm/DBS] DBS pypi packaging (#591)

fine with being careful, i am simply a bit lost as to what the goal is and how this impacts CRAB

On 19/02/2019 10:32, Valentin Kuznetsov wrote:

Stefano, as I reported on meeting, we should be careful with installations until proper license will be in place. Best, Valentin.

On 0, Stefano Belforte notifications@github.com wrote:

It may be off topic, but I find ti REALLY GREAT that now I can run dbs client on my Ubuntu laptop ! (latest Ubuntu has proper libcurl out of the box).

I am interested also in wmcore, I see that pip has 1.1.1rc7, do you plan to make and keep that up to date ?

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DBS/pull/591#issuecomment-465201810https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465201810&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=aYAUobOHc99ofOip-rRR49snFUGN1RDkOZ5EMOKv6no&e=

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmwm/DBS/pull/591#issuecomment-465207134https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465207134-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=2ooGSj0gpAw2Ptka9Ye-KV4atbuEyAeqgqK8fKNgK08&e=, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwK2adbW4DxSie2OGaPcZ7QGiA0XUyRks5vPCcXgaJpZM4Z8HPHhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABwK2adbW4DxSie2OGaPcZ7QGiA0XUyRks5vPCcXgaJpZM4Z8HPH-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=h_R4nqSKTgFUIN7dGZ7gETpqmPokBXtiPjSCOTD5f8w&e=.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465220822&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=vY7sHw9hIaeTJ0bumpGSbvbKIy75PwSszdSJ9y6LyiY&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABsXTiuS473IEbNxNzWEoBwZpQlt3Yrbks5vPC5YgaJpZM4Z8HPH&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=aJPX5wVS63cMoA3fPmI_dP7X9zw5l-XF6z2zEYA8X4A&e=.

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DBS/pull/591#issuecomment-465224902

yuyiguo commented 5 years ago

Thanks for clarifying things. There was some miss information I got before. Yuyi

From: Valentin Kuznetsov notifications@github.com Reply-To: dmwm/DBS reply@reply.github.com Date: Tuesday, February 19, 2019 at 11:28 AM To: dmwm/DBS DBS@noreply.github.com Cc: Yuyi Guo yuyi@fnal.gov, Mention mention@noreply.github.com Subject: Re: [dmwm/DBS] DBS pypi packaging (#591)

Yuyi, on a WMA meeting we discussed this. Original goal was to simplify WMA deployment and not a public release of DBS client.

Regarding license, you need to understand that whatever you release to public domain (in this case pypi) can be used for different purposes. All CMS software is open source, but according to this note:

https://indico.cern.ch/event/791214/attachments/1783962/2906417/CBnewsletter-CB122-February-2019.pdf

the collaboration board discuss a possibility to release all CMS software under apache 2 license instead of GPL one (may be it is default one), see item 7.

I suggested on a meeting that our release of software to public should be adopted with certain license and I asked Alan to clarify what it should be.

As developers we should be aware of how our software is distributed to avoid possible issues. That's why I made my suggestions that some license should be added to pypi packages.

Best, Valentin.

On 0, Yuyi Guo notifications@github.com wrote:

The same question here. What is the goal of all this? What kind license do I need ? But it is still a good news that Stefano can run it out of box w/o CMS specific software decency. Can we improve DBS pypi package to get the newest libcurl? Thanks, Yuyi

From: Stefano Belforte notifications@github.com Reply-To: dmwm/DBS reply@reply.github.com Date: Tuesday, February 19, 2019 at 11:03 AM To: dmwm/DBS DBS@noreply.github.com Cc: Yuyi Guo yuyi@fnal.gov, Mention mention@noreply.github.com Subject: Re: [dmwm/DBS] DBS pypi packaging (#591)

fine with being careful, i am simply a bit lost as to what the goal is and how this impacts CRAB

On 19/02/2019 10:32, Valentin Kuznetsov wrote:

Stefano, as I reported on meeting, we should be careful with installations until proper license will be in place. Best, Valentin.

On 0, Stefano Belforte notifications@github.com wrote:

It may be off topic, but I find ti REALLY GREAT that now I can run dbs client on my Ubuntu laptop ! (latest Ubuntu has proper libcurl out of the box).

I am interested also in wmcore, I see that pip has 1.1.1rc7, do you plan to make and keep that up to date ?

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DBS/pull/591#issuecomment-465201810https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465201810&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=aYAUobOHc99ofOip-rRR49snFUGN1RDkOZ5EMOKv6no&e=https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465201810-253Chttps-3A__urldefense.proofpoint.com_v2_url-3Fu-3Dhttps-2D3A-5F-5Fgithub.com-5Fdmwm-5FDBS-5Fpull-5F591-2D23issuecomment-2D2D465201810-26d-3DDwQFaQ-26c-3DgRgGjJ3BkIsb5y6s49QqsA-26r-3D8bursUuc0V63OwREQMBG2Q-26m-3DiWTqQw-2D66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc-26s-3DaYAUobOHc99ofOip-2DrRR49snFUGN1RDkOZ5EMOKv6no-26e-3D-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=minQC_zTQjQM4uditoRQipQaRZfIGg87Uoz9KopaF1w&s=p63JaJ9wsAMGyAgNHA6HQmUn-9hE4EVJEndN7ArEQWg&e=

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmwm/DBS/pull/591#issuecomment-465207134https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465207134-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=2ooGSj0gpAw2Ptka9Ye-KV4atbuEyAeqgqK8fKNgK08&e=https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465207134-253E-253Chttps-3A__urldefense.proofpoint.com_v2_url-3Fu-3Dhttps-2D3A-5F-5Fgithub.com-5Fdmwm-5FDBS-5Fpull-5F591-2D23issuecomment-2D2D465207134-2D253E-26d-3DDwQFaQ-26c-3DgRgGjJ3BkIsb5y6s49QqsA-26r-3D8bursUuc0V63OwREQMBG2Q-26m-3DiWTqQw-2D66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc-26s-3D2ooGSj0gpAw2Ptka9Ye-2DKV4atbuEyAeqgqK8fKNgK08-26e-3D-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=minQC_zTQjQM4uditoRQipQaRZfIGg87Uoz9KopaF1w&s=6p4gNaeozndnDVQJVEx_0Jd7AzxDxmwN8vakXXWHjec&e=, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwK2adbW4DxSie2OGaPcZ7QGiA0XUyRks5vPCcXgaJpZM4Z8HPHhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABwK2adbW4DxSie2OGaPcZ7QGiA0XUyRks5vPCcXgaJpZM4Z8HPH-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=h_R4nqSKTgFUIN7dGZ7gETpqmPokBXtiPjSCOTD5f8w&e=https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABwK2adbW4DxSie2OGaPcZ7QGiA0XUyRks5vPCcXgaJpZM4Z8HPH-253E-253Chttps-3A__urldefense.proofpoint.com_v2_url-3Fu-3Dhttps-2D3A-5F-5Fgithub.com-5Fnotifications-5Funsubscribe-2D2Dauth-5FABwK2adbW4DxSie2OGaPcZ7QGiA0XUyRks5vPCcXgaJpZM4Z8HPH-2D253E-26d-3DDwQFaQ-26c-3DgRgGjJ3BkIsb5y6s49QqsA-26r-3D8bursUuc0V63OwREQMBG2Q-26m-3DiWTqQw-2D66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc-26s-3Dh-5FR4nqSKTgFUIN7dGZ7gETpqmPokBXtiPjSCOTD5f8w-26e-3D-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=minQC_zTQjQM4uditoRQipQaRZfIGg87Uoz9KopaF1w&s=dFLbpG1MqQ3CYTGGW0ucDl_9lxPVT19EhCafWaT901c&e=.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465220822&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=vY7sHw9hIaeTJ0bumpGSbvbKIy75PwSszdSJ9y6LyiY&e=%3E, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABsXTiuS473IEbNxNzWEoBwZpQlt3Yrbks5vPC5YgaJpZM4Z8HPH&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=iWTqQw-66O68zgC4cKBsEvDu5Lzmn7Hkiw394PaORDc&s=aJPX5wVS63cMoA3fPmI_dP7X9zw5l-XF6z2zEYA8X4A&e=%3E.

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DBS/pull/591#issuecomment-465224902https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465224902&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=minQC_zTQjQM4uditoRQipQaRZfIGg87Uoz9KopaF1w&s=T2vL4_Ucu2kF59BFZP0CKJhJHPNnmeFOwfvAoWsuFJ0&e=

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_DBS_pull_591-23issuecomment-2D465230518&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=minQC_zTQjQM4uditoRQipQaRZfIGg87Uoz9KopaF1w&s=bC7GXR8bTuto37A2nx36vrY-2LbLgi3G5K0BU_5j3ZA&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABsXTmUDl0jeRCPsPtcu-5FotFb0SZREU5ks5vPDRBgaJpZM4Z8HPH&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=8bursUuc0V63OwREQMBG2Q&m=minQC_zTQjQM4uditoRQipQaRZfIGg87Uoz9KopaF1w&s=CZOVDFieGszm0ADyiLQ260UkFe3eF3wwGsB78TQ9PNo&e=.

amaltaro commented 5 years ago

Yuyi, on a WMA meeting we discussed this. Original goal was to simplify WMA deployment and not a public release of DBS client.

if you want to simplify the deployment procedure by using Pypi packages instead of RPM, how can you possibly not make it available to the general public?!?

As I said before a couple of times, and so did Erik, the main goal of these developments is to be able to deploy WMAgent via Pypi. If someone else installs it, then it's on their own.

About the license, sure, let's discuss it and get it properly done with the CMS packages. However, what's out there is already there and there is nothing that we can do but discussing the correct license and making the necessary changes. I don't think the license thing is a show stopper to proceed with this PR.

Yuyi, if you feel there is anything else missing here, please let us know such that we can move forward.

yuyiguo commented 5 years ago

Alan, I did not know about the goal of making DBS client in pypi or the license issue involved so I asked. Valentin's reply made sense. What I said stopped you moving forward? If so, I might be miss understood.

Stefano's comment regarding the newest libcurl could be something to make the package to be used out of box. So I ask if it can be improved. What you think? Thanks, Yuyi

yuyiguo commented 5 years ago

@belforte Hi Stefano, Could you let us know the version of libcurl in your new Ubuntu machine? Thanks, Yuyi

belforte commented 5 years ago

I wish I knew how tpo answer I had to install (ana)conda to get xroot client and now it overrode things, so I have:

LapSB:~$ which curl /home/belforte/mybin/anaconda2/bin/curl LapSB:~$ curl --version curl 7.63.0 (x86_64-conda_cos6-linux-gnu) libcurl/7.63.0 OpenSSL/1.0.2p zlib/1.2.11 libssh2/1.8.0 Release-Date: 2018-12-12 Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy LapSB:~$

while in standard place I have: /usr/lib/x86_64-linux-gnu/libcurl.so.4 /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0

and if you want some details about it, I do not now how to get them .

I do not know how to go back to non-conda shell. Anyhow it was just ubuntu 18.04 LTS and curl have been always working for me.

vkuznet commented 5 years ago

Stefano, to go out from conda environment you should deactivate it: conda deactivate

Then Ubuntu is debian based Linux distro, so it uses apt tools. You can list installed packages as easy as following: apt list --installed

Best, Valentin.

On 0, Stefano Belforte notifications@github.com wrote:

I wish I knew how tpo answer I had to install (ana)conda to get xroot client and now it overrode things, so I have:

LapSB:~$ which curl /home/belforte/mybin/anaconda2/bin/curl LapSB:~$ curl --version curl 7.63.0 (x86_64-conda_cos6-linux-gnu) libcurl/7.63.0 OpenSSL/1.0.2p zlib/1.2.11 libssh2/1.8.0 Release-Date: 2018-12-12 Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy LapSB:~$

while in standard place I have: /usr/lib/x86_64-linux-gnu/libcurl.so.4 /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0

and if you want some details about it, I do not now how to get them ..

I do not know how to go back to non-conda shell. Anyhow it was just ubuntu 18.04 LTS and curl have been always working for me.

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DBS/pull/591#issuecomment-465738405

belforte commented 5 years ago

thanks Valentin , so here's my setup , hope it is what Yuyi wants: LapSB:~$ which curl /usr/bin/curl LapSB:~$ curl --version curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.0g zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3 Release-Date: 2018-01-24 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL LapSB:~$ LapSB:WMCore$ apt list --installed|grep curl

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

curl/bionic-updates,bionic-security,now 7.58.0-2ubuntu3.6 amd64 [installed,automatic] libcurl3-gnutls/bionic-updates,bionic-security,now 7.58.0-2ubuntu3.6 amd64 [installed] libcurl4/bionic-updates,bionic-security,now 7.58.0-2ubuntu3.6 amd64 [installed,automatic] LapSB:WMCore$ which curl /home/belforte/mybin/anaconda2/bin/curl LapSB:WMCore$