Improve the package-ability of hydroffice.bag

[ghisvail]

Following our discussion started on the hdf-compass issue tracker, I made an attempt to package hydroffice.bag for Debian but hit a few issues. The package overall builds fine with the current Python 2 and Python 3 stack on Debian, but its final content does not meet the standard of our packaging policies.

The main problem concerns the package_data, which are installed under the Python system site-packages path (/usr/lib/pythonX.Y/dist-packages/...) instead of the recommended architecture independent path (/usr/share/...). The code in the helper module does not apparently handle arbitrary paths (only paths relative to the module file) so just patching setup.py to install the files listed in package_data somewhere else won't work.

On my end, the goal is to achieve a binary packaging organized the following way:

• a Python 2 package to use the hydroffice.bag library with the Python 2 interpreter
• a Python 3 package to use the hydroffice.bag library with the Python 3 interpreter
• a "common" package containing the package_data files shared between the Python 2 and 3 packages
• a "doc" package containing the HTML documentation
• a "tools" package containing the command-line scripts

Thanks for your help.

[giumas]

@ghisvail Thank you for the suggestions! Do you have some hints to existing compliant packages that have similar requirements (a package_data) so that I can follow their approach?

I will work on them during next week (together with some other planned improvements). I will contact you as soon as I have a 2.7 RC to test if it is compliant with the Debian policy.

[giumas]

@ghisvail I looked around to find Python packages with such an architecture to see how to modify the setup.py.. but I didn't have luck. I understand the advantages of the structure that you describe.. but I am not sure how to properly implement it.

For instance, this is currently a pure Python library that works with both Python 2 and 3 without six. So what do you mean with a Python 2 package [for] the Python 2 interpreter, and a Python 3 package [for] the Python 3 interpreter?

BTW, I am also attempting to follow: https://python-packaging-user-guide.readthedocs.org/en/latest/

[ghisvail]

I am not sure how to properly implement it.

I can work around it with symlinks. Please disregard this specific comment.

So what do you mean with a Python 2 package [for] the Python 2 interpreter, and a Python 3 package [for] the Python 3 interpreter?

Means one package installs h.b in the system Python 2 dist-packages, and the other package in the system Python 3 site-packages.

BTW, I am also attempting to follow: https://python-packaging-user-guide.readthedocs.org/en/latest/

These are good practices indeed.

[ghisvail]

Can I rely on this repository to sync the packaging with, or shall I use the tarballs on PyPI instead?

[giumas]

Pypi wheels and github tag releases should be in sync. Drop me a line if you discover any issue on that.

[ghisvail]

setup_args['install_requires'] =\
    [
        "lxml",
        "numpy",
        "gdal<2.0",
    ]

Debian has gdal >= 2.0. Any plans to support this version?

[giumas]

The issue with gdal>=2.0 is strictly related to the freezing:

• https://github.com/pyinstaller/pyinstaller/issues/1522
• https://trac.osgeo.org/gdal/ticket/6364

As gdal 2.1.0 that fixes the issue has been released a few hours ago: https://pypi.python.org/pypi/GDAL/2.1.0 , the question really would be: any plan for Debian to support gdal 2.1.0 ?

[ghisvail]

The issue with gdal>=2.0 is strictly related to the freezing

Since I don't use pyinstaller, it should be fine then?

any plan for Debian to support gdal 2.1.0 ?

The Debian GIS team is quite active, so I expect gdal 2.1.0 to come really soon.

[giumas]

New tag without the strict GDAL requirement: https://github.com/hydroffice/hyo_bag/releases/tag/v0.2.12

[ghisvail]

Thanks

[ghisvail]

Ok, now I have a candidate package based on v0.2.12.

[giumas]

Great!

[ghisvail]

What is the purpose of the media/* files listed in package_data?

[giumas]

To install the library icons

[ghisvail]

Are you planning to provide source tarballs on PyPI alongside the wheels in the future? Or should I rely on Github's ones?

[ghisvail]

One big advantage (for us) in providing a tarball on PyPI is that the source can be curated from the useless (for us) CI scripts and VCS files.

[giumas]

I think that the PyPi solution should be always preferred: https://pypi.python.org/pypi/hydroffice.bag Let me know if the tarball meets your requirements

[ghisvail]

Well, the docs and tests are missing, which means neither a documentation package can be built nor any testing can be performed. For the rest (examples and specs), it depends whether you think they would be worth providing to the end-user, perhaps in a separate package. Your call.

[giumas]

I have added docs, tests, and examples. I don't see the point to add specs. Give it a look..

[ghisvail]

Looks good to me, I'll report if I am missing anything else. Packaging coming soon.

[ghisvail]

The tarball is missing AUTHORS.rst and HISTORY.rst required by setup.py to build the long description.

[giumas]

Corrected. Thank you!

[ghisvail]

Ok the last warnings I get from the packaging come from the absence of manpage for the console scripts. I'll attempt a submission without those, since I don't feel qualified to write them. However, it would be nice to think about generating these manpages later.

There are multiple options for solving this:

• Use help2man in a custom build command. (for instance in obspy)
• Write them from scratch with sphinx.
• Use a sphinx plugin such as autoprogram.

The last option is probably interesting for you, since all your scripts use a parser from argparse.

[giumas]

Let me know if the submission is rejected for the missing manpages. However, we will most likely adopted in a short time the autoprogram solution given its simplicity.

[ghisvail]

hydroffice.bag is now available in Debian.

[ghisvail]

https://tracker.debian.org/pkg/hydroffice.bag

[giumas]

Great!