Empty dependency `__pycache__` folders left behind on upgrade

[legoktm]

Description

When upgrading from 2.5.1 to 2.6.0~rc1 (current develop), dpkg emits warnings that it can't remove the passlib directories because of the __pycache__ directories.

Steps to Reproduce

• Install 2.5.1 on prod/staging
• make build-debs on develop
• Install the new securedrop-app-code deb

Expected Behavior

No dpkg warnings; /opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ is deleted

Actual Behavior

Unpacking securedrop-app-code (2.6.0~rc1+focal) over (2.5.1+focal) ...                                                 
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/compat/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/compat': Directory not empty                                                                                          
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/tests/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/tests': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/handlers/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/handlers': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/django/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/django': Directory not empty                                                                                            
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/scrypt/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/scrypt': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/_blowfish/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/_blowfish': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/__pycache__': Directory not empty                                                                                    
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/_setup/__pycache__': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/_setup': Directory not empty
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/__pycache__': Directory not empty                                                                                           
dpkg: warning: unable to delete old directory '/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib': Directory not empty

Comments

All of the code is correctly removed, it's just the empty folders left behind...

sdadmin@app:~$ find /opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/compat
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/compat/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/utils/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/tests
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/tests/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/scrypt
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/scrypt/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/_blowfish
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/crypto/_blowfish/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/django
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/ext/django/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/_setup
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/_setup/__pycache__
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/handlers
/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/passlib/handlers/__pycache__

[legoktm]

To clarify, there is no actual problem caused by this, it's just messy and an unintentional consequence of having the postinst generate the __pycache__ folders.

[legoktm]

Looking at what dh-python does, we need a prerm script that runs py3clean; see https://salsa.debian.org/python-team/tools/dh-python/-/blob/master/autoscripts/prerm-py3clean

So the flow ends up being:

• first install triggers postinst - runs py3compile to generate cache
• upgrade triggers:
  • prerm (before new version is unpacked) - runs py3clean to delete existing cache
  • postinst (after new version is in place) - runs py3 compile to generate cache from scratch

[legoktm]

[Backlog pruning, 5/15] Removing this from 2.6.0, it's not important enough.