Missing setup.py for dbs3-client==4.0.1

[todor-ivanov]

This issue is related mostly to the package versions 4.0.0 and 4.0.1 While testing the latest dbs3-client version for WMCore (in a custom virtual environment) [1] we found that due to the missing setup.py file the installation fails with the standard command:

pip install dbs3-clinet==4.0.1

This can easily be prevented if setup.py is provided or linked to setupPypi.py in the tar file.

FYI @vkuznet @yuyiguo

[1] https://github.com/dmwm/WMCore/issues/10884#issuecomment-989859110

[vkuznet]

@yuyiguo could you please resolve it. I didn't see setup.py in previous versions either and I didn't touch explicitly this problem. If you want me to fix it that's fine, but I want your word first this you originally put files into this repo and maintain it, and I don't know full logic of setupPyPi.py appearance.

[yuyiguo]

@todor-ivanov @vkuznet The setup for pypi is https://github.com/dmwm/DBSClient/blob/main/setupPypi.py.
I did pip install dbs3-client==3.17.4 and had no problem. Please see the instructions on how to install DBSClient. As I recall, WM team used pypi install DBSClient in the past too.

Valentin, if you want to add link or change the file name, please feel free to do so. If the file name is changed, please update the wiki page.

[vkuznet]

@yuyiguo the issue with install that for versions 3.X.X you have whl package n PypPi and this is what pip is using, see

pip3 install --user dbs3-client==3.17.4
Collecting dbs3-client==3.17.4
  Downloading dbs3_client-3.17.4-py3-none-any.whl (19 kB)
Collecting pycurl==7.43.0.6
  Downloading pycurl-7.43.0.6.tar.gz (222 kB)
     |████████████████████████████████| 222 kB 2.3 MB/s
  Preparing metadata (setup.py) ... done
Collecting dbs3-pycurl
  Downloading dbs3_pycurl-3.17.4-py3-none-any.whl (11 kB)
Using legacy 'setup.py install' for pycurl, since package 'wheel' is not installed.
Installing collected packages: pycurl, dbs3-pycurl, dbs3-client
    Running setup.py install for pycurl ... done
Successfully installed dbs3-client-3.17.4 dbs3-pycurl-3.17.4 pycurl-7.43.0.6

The question is how do you create it in PyPi. I only used twine to upload the new package version, but it only creates the tar ball. And, since dbs3-client tar ball does not have setup.py it fails to be installed, see

pip3 install dbs3-client==4.0.1

Collecting dbs3-client==4.0.1
  Downloading dbs3-client-4.0.1.tar.gz (24 kB)
ERROR: File 'setup.py' not found for legacy project dbs3-client==4.0.1 from https://files.pythonhosted.org/packages/ff/d4/f7ab682fc1a7a89276b31977b3eea64685475567aaa83b90ca52fc392363/dbs3-client-4.0.1.tar.gz#sha256=117dcddca9168198e86fcae670b3cd526fccbd2fd415fe18540a0a50cab243b4.

In your wiki there is not instructions how to you upload the package to Pypi and this is the root of the problem, since I used twine and it does not seem to be able to create whl package. Please elaborate how you upload the code to PyPi. Meanwhile, setup.py is a standard way to setup things in Python and if you agree we should only rename setupPyPi.py to setup.py to make things work.

[vkuznet]

For clarity here is how I uploaded dbs3-client:

# remove everything in dist if it exists
rm dist/*

# prepare tar ball
python3 setupPypi.py sdist

# upload tar ball to PypI
twine upload --username <USERNAME> -p <PASSWORD> --verbose dist/*

[amaltaro]

Perhaps what you need to do when packaging those is:

mv setupPypi.py setup.py
python3 -m build

? This should build both packages (tar.gz and wheels)

[yuyiguo]

@vkuznet the setup file name can be anything you want. You just need to build with it.

python3 setupPypi.py sdist
python3 setupPypi.py bdist_wheel
python3 -m twine upload --repository pypi dist/*

All these are documented in wiki in details. Please see https://github.com/dmwm/DBSClient/wiki/Upload-to-pypi .

[vkuznet]

@yuyiguo (sorry my keyboard start glitching). I don't think the instructions are correct as I can't do bdist:

python3 setupPypi.py bdist_wheel
usage: setupPypi.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setupPypi.py --help [cmd1 cmd2 ...]
   or: setupPypi.py --help-commands
   or: setupPypi.py cmd --help

error: invalid command 'bdist_wheel'

[vkuznet]

and, as far as I know, the standard way to setup file is to call it setup.py and not as anything we like since python tools like pip use that specific setup.py name to install things.

[vkuznet]

@amaltaro and your method does not work either:

python3 -m build
/opt/local/bin/python3: No module named build.__main__; 'build' is a package and cannot be directly executed

So, it really depends how setup.py is written. Again, I did not create the package and I don't know how originally it was using to build stuff, that's why I'm asking questions.

[yuyiguo]

@vkuznet We had to support both pypi and rpm in the past. So we have two different setup files for each. Now we dropped the rpm support. I am Ok if you want to change it to setup.py. What I wanted to point out was that you had something wrong with your build env. This was nothing to do with the setup file name. I was able to build and deploy multiple versions with the setupPypi.py file. DBS clients were able to install it. The problem only showed up when you build 4.0.*. I was surprised you did not test it after uploading to pypi.

[vkuznet]

@yuyiguo , as I wrote, I didn't maintain the package, neither I created all these files. What you describe obviously works on your system but not on mine. We can argue which setup is right or wrong, but in fact it is irrelevant since python has numerous ways to install stuff and all of them should be correctly written in its setup file. As such a different python setup on one system may not coincide with another.

Bottom line, I copied setup.py, created new tag, uploaded it with twine using sdist and now it works. I don't know how in fact you build bdist_wheel since nothing in setup has reference to it. And, it may be that on your system you have additional wheel related setup tools which on mine I don't have, which will explain why our systems are differ. Again, neither env is right or wrong, they are different.

@todor-ivanov please try out version 4.0.2. Here is what I see now in my custom install:

pip3 install --user dbs3-client==4.0.2
Collecting dbs3-client==4.0.2
  Downloading dbs3-client-4.0.2.tar.gz (24 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: pycurl==7.43.0.6 in /Users/vk/Library/Python/3.9/lib/python/site-packages (from dbs3-client==4.0.2) (7.43.0.6)
Collecting dbs3-pycurl==3.17.4
  Using cached dbs3_pycurl-3.17.4-py3-none-any.whl (11 kB)
Using legacy 'setup.py install' for dbs3-client, since package 'wheel' is not installed.
Installing collected packages: dbs3-pycurl, dbs3-client
    Running setup.py install for dbs3-client ... done
Successfully installed dbs3-client-4.0.2 dbs3-pycurl-3.17.4

[yuyiguo]

@vkuznet Glad it started to work for you. Can you update the wiki to reflect the new setup file?

[vkuznet]

@yuyiguo , quick look-up on internet reveals that in order to use bdist_wheel someone needs to install wheel package on its system. This explain why our system is different. And, as I did it, I was able to run python3 setup.py bdist_wheel. Therefore, the wiki must be updated to let users know that wheel package should be installed on his/her system in order to use it for builds. It just confirms that Python setup is far from trivial and on every system is should be carefully tested and all dependencies should be in place to work together. I'll update wiki with that info.

[vkuznet]

@yuyiguo wiki is updated, and I uploaded wheel package of current release.

[yuyiguo]

@vkuznet Thanks for updating the wiki. We tried to document as much as we can, but missing something happens especially when there are a lot of new stuff.

[yuyiguo]

Done

[todor-ivanov]

Hi All,

@todor-ivanov please try out version 4.0.2. Here is what I see now in my custom install:

I can confirm the version 4.0.2 is working perfectly now. Thank you @vkuznet @yuyiguo !!!