spotify / dh-virtualenv

Python virtualenvs in Debian packages
http://dh-virtualenv.readthedocs.io/en/latest/
GNU General Public License v2.0
1.62k stars 185 forks source link

Clearing out .pyc files and __pycache__ directories on package removal #270

Open wwuck opened 5 years ago

wwuck commented 5 years ago

I looked at Issue #45 where it's mentioned why .pyc files are bundled in with the .deb package. I ran into a case where I was installing an updated version of a .deb package generated with dh-virtualenv that had removed some of the python package requirements that existed in the previous .deb package. This generated some dpkg warnings when installing the new .deb file over the old package version.

dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/requests/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/requests': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/idna/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/idna': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/utils/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/utils': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/transforms/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/transforms': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/schemas/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/schemas': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/model/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/model': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/grid/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/grid': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/estimators/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/estimators': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/backend/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/backend': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/mypackage/lib/python3.5/site-packages/h2o': Directory not empty

Is it possible for dh-virtualenv to generate a prerm script that will clear out these old files and directories?

It looks like dh_python2 and dh_python3 use the prerm-pyclean and prerm-py3clean autoscripts to do this. Could this be done with dh-virtualenv too? Is there any way to add these manually into the debian/ directory, or does this have to be activated from within dh-virtualenv?

/usr/share/debhelper/autoscripts/prerm-pyclean
/usr/share/debhelper/autoscripts/prerm-py3clean

Also, I notice there are autoscripts for postinst-pycompile and postinst-py3compile, so those could also be used instead of bundling the .pyc files in the .deb?

/usr/share/debhelper/autoscripts/postinst-pycompile
/usr/share/debhelper/autoscripts/postinst-py3compile
mawiegand commented 5 years ago

I had a similar problem. My package couldn't be removed completely because of existing .pyc files and __pycache__ directories.

I temporarily solved it by adding py3clean . to override_dh_virtualenv and providing a prerm script to call py3clean . before the package is removed.

jhermann commented 5 years ago

For reference:

$ dpkg -S py3clean
python3-minimal: /usr/bin/py3clean
dh-python: /usr/share/debhelper/autoscripts/prerm-py3clean
python3-minimal: /usr/share/man/man1/py3clean.1.gz
$ cat /usr/share/debhelper/autoscripts/prerm-py3clean
if which py3clean >/dev/null 2>&1; then
    py3clean -p #PACKAGE# #ARGS#
else
    dpkg -L #PACKAGE# | perl -ne 's,/([^/]*)\.py$,/__pycache__/\1.*, or next; unlink $_ or die $! foreach glob($_)'
    find /usr/lib/python3/dist-packages/ -type d -name __pycache__ -empty -print0 | xargs --null --no-run-if-empty rmdir
fi
wwuck commented 5 years ago

Don't forget there is also a prerm-pypyclean and a postinst-pypycompile, if dh-virtualenv supports pypy.

nailor commented 5 years ago

Hi!

Thanks for the bug report. Feels definitely something we should work on. I wonder what is the general opinion on how to handle this: We could default on some py*clean step to the build chain, but unsure if we could target the right python version?

In any case it should be overridable

jhermann commented 5 years ago

It is not a build thing, but a maintainer script thing. As to versions, the helper scripts have to be called explicitly with the venv binary, and the venv path – from debhelper snippets.

wwuck commented 5 years ago

I attempted to add the following to a prerm script as a quick way to get this functionality, but I'm not sure how I embed the value of $DH_VIRTUALENV_INSTALL_ROOT into the prerm script.

${DH_VIRTUALENV_INSTALL_ROOT}/<package-name>/bin/python3 /usr/bin/py3clean ${DH_VIRTUALENV_INSTALL_ROOT}/<package-name>

Any suggestions?

EDIT: I guess I could just hardcode the value manually as a quick fix.

jhermann commented 5 years ago

you place your code under the #debhelper# thing.