Py3 dbs3 client - py3 codechanges

[todor-ivanov]

DO NOT MERGE YET

As discussed in a meeting we had between me, @amaltaro and @mapellidario this PR was created in order to separate the Py3 code changes, as they have advanced so far, from the basics changes to the setup script needed for the packages to build properly.

The current PR provides the python3 related changes. The old PR provides the minimal changes in setup.py.

p.s. Currently the PR is created against Master, but once we have the new branch created I will change it here too.

[amaltaro]

@todor-ivanov I had the impression we agreed in dropping the jsonwrapper and simply keep the standard library json import, no?

Just for convenience, I am going to paste here the course of actions we came up with this morning:

• Alan: if Yuyi agrees, ask her to create a new branch for the python3-client changes
• Todor: once the branch is created, provide the setup.py changes in a separate PR (make sure to make it against the new branch).
  • PS: you might want to run 2to3 on this setup.py to convert it to python3 as well.
• Todor: run 2to3 on the dbs3-client package and provide those python3-only changes in a new PR (against the new py3 branch)
• Todor: run 2to3 on the dbs3-pycurl-client package and provide those python3-only changes in a new PR (against the new py3 branch)
• Yuyi: merges all these and create a new tag in DBS
• Todor: create a WMCore PR updating the requirements_py3 text file.

[yuyiguo]

@amaltaro @todor-ivanov

I ran 2to3 on entire DBS code suite. I committed them into a py2to3 branch. You may start with that.

[yuyiguo]

@amaltaro @todor-ivanov Per slack conversion with Alan, I created another branch py3-4-wm that is the same as the current prod w/o running 2to3.

[todor-ivanov]

Hi @vkuznet, @yuyiguo,
Since we decided to keep all the changes regarding the DBS Client migration to Python3 in a separate branch - I see no point of keeping the jsonwrapper around any more. Plus it was giving me hard time because in cjson.decode there is a quite general approach to the input object (if it is a string or a file descriptor), which had to be reflected into the jsonwrapper etc. etc. ... I see no point of keep adding more and more lines when at the end it comes to a change as simple as substituting few lines (literally speaking) [1].

[1] https://github.com/dmwm/DBS/pull/653/commits/c565ac91432cfcec258226a397bc03b69ce335f5#diff-ff347d12d7613a29156638970b91a48ffe08bece319648e497bc1101d23417e6

[todor-ivanov]

I created another branch py3-4-wm that is the same as the current prod w/o running 2to3.

Thanks @yuyiguo, I started the work towards the fresh branch py3-4-wm With my last commit I've just cut the backwards compatibility to py2: https://github.com/dmwm/DBS/pull/653/commits/d6aba693ff8274acd83e1efca355651d9e66a625

[yuyiguo]

@todor-ivanov Move away from the py2 compatibility looks like a good choice to me.

[todor-ivanov]

Thanks @yuyiguo I think I just completed the transition with my latest commit: https://github.com/dmwm/DBS/pull/653/commits/cce4d22d49bd54f26b305fe99cc0aa89b4268091

It was a must to move from StringIO to BytesIO buffers, since everywhere in the pycurl documentation it is stressed that the return values from major methods like Curl.perform() or Curl.getinfo() are actually different for py2 and py3: [1]

And I just managed to run one basic API [2] @yuyiguo it would be nice if you can help with validating the rest of the APIs, even though we expect no complications there.

[1]

Help on class Curl in module pycurl:
class Curl(builtins.object)
...
getinfo_raw(...)
return a Python byte string (``str`` on Python 2, ``bytes`` on Python 3).
etc.

[2]

In [6]: from dbs.apis.dbsClient import DbsApi
In [7]: dbsApi = DbsApi(url = u'https://cmsweb-testbed.cern.ch/dbs/int/global/DBSReader')
In [8]: dbsApi.listDataTiers()
Out[8]: 
[{'create_by': 'yuyi',
  'creation_date': 1398704682,
  'data_tier_id': 31517,
  'data_tier_name': 'DQMIO'},
 {'create_by': 'vlimant',
  'creation_date': 1509096851,
  'data_tier_id': 2056894,
  'data_tier_name': 'NANOEDMAOD'},
...
]

[yuyiguo]

@todor-ivanov I hope I can look into the APIs next Monday. As I explained in the email, I am working on some urgent PII stuff.

[yuyiguo]

@todor-ivanov @amaltaro Meged todor-ivanov:py3-dbs3-client-codechanges into py3-4-wm

[todor-ivanov]

Thank @yuyiguo

[amaltaro]

@yuyiguo Hi Yuyi, could you please also create a new tag including these changes (thus pointing to this branch)? Then we can update our WMCore PY3 specs to point to the official repository instead of Todor's remote.

[yuyiguo]

@amaltaro @todor-ivanov

The last production DBS tag was 3.16.0. I created tag 3.16.1000 for my development of py3. I will use the 3.16.100x for my versions. I am thinking for all the client development we will use 3.16.200x. So we can each follow different tag scheme and merge them into something like 3.17.x later. I hope this will not cause any confusion. Please let me know if this is fine with you guys. I will create tag as soon as I hear from you.

[amaltaro]

@yuyiguo this scheme looks good to me. Please let me know when the python3 client is tagged with Todor's changes.

[yuyiguo]

@amaltaro 3.16.2000 tag created : https://github.com/dmwm/DBS/releases/tag/3.16.2000