Dependencies, and updating setup.py

[sde1000]

I've been looking at making a new release (upload to pypi, and generate Debian packages) and I've run into a couple of blockers.

When I try to build the Debian packages with current debhelper, I get:

   dh_auto_test -O--buildsystem=pybuild
I: pybuild base:217: python3.8 setup.py test 
running test
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
error: the `allow-hosts` option is not supported when using pip to install requirements.
E: pybuild pybuild:341: test: plugin distutils failed with: exit code=1: python3.8 setup.py test 

What's happening here is that debhelper is deliberately setting up the environment to prevent setuptools from fetching build or test dependencies. Furthermore, reading around the issue, I've found that setup.py test is deprecated: see https://github.com/pypa/setuptools/issues/1684 — the reasoning appears to be that, amongst other things, fetching and installing test requirements is out of scope for setuptools.

Currently in setup.py we have:

    test_suite='dali.tests',
    install_requires=['pyusb', 'pymodbus'],
    tests_require=['mock'],

This works well enough for the current Travis CI integration, but is obviously going to be problematic in the future. There are a couple of things to talk about:

• I'm not particularly happy with this library having dependencies other than the standard Python libraries. The core of the library has no dependencies.
• I don't really want to have to declare pyusb and pymodbus as Build-Depends in the Debian control file. (Or have debhelper add these Depends to the built package.)

If I remove those three lines from setup.py then debhelper builds the packages without running any tests. If I add dali.tests to the packages list then debhelper runs all the tests using python3 -m unittest discover — which fails on the unipi driver tests which require mock and pymodbus.

So what should we do? I'm not sure, but things I'm considering include:

• move the unipi tests (and any future driver tests) to a separate package, probably under dali.driver, and arrange that they are not run by default by python3 -m unittest discover (but are run by Travis CI)
• possibly split drivers into a completely separate project?
• possibly add configuration for something like tox to run all the tests with their dependencies installed? Apparently this doesn't play nicely with Travis CI, though. Why not both, other than the additional maintenance effort?

Any other suggestions? Python packaging seems to be a bit of a minefield at the moment!

[dgomes]

You can run tox + travis, it was the CI in Home Assistant for a long time (currently it has moved to Azure pipelines)

I would prefer we don't split the project, it is not such a big project to start splitting.

[mhemeryck]

Any other suggestions? Python packaging seems to be a bit of a minefield at the moment!

Always has been :)

You could perhaps use something like setuptools extra to declare extra dependencies.

About the debian stuff: isn't it possible to move that out of the repo and make the generation of it completely part of the CI? (I don't have any experience with debian packaging though).

[sde1000]

Oh, yes, that looks very useful. Declare the dependencies for each driver as extras that are only required if you want to use that driver.

Debian stuff: it's complicated. The usual Debian model is that there's a separation between "upstream" packages and their Debian packaging; the packaging is itself a project, with its own maintainer, repo, etc.

For small programs like this one that haven't been included in Debian, this separation is a lot of hassle and so I just include the control files and scripts to build Debian packages in the repo.