canonical / ubuntu-pro-client

Ubuntu Pro Client for offerings from Canonical
https://canonical-ubuntu-pro-client.readthedocs-hosted.com/en/latest/
GNU General Public License v3.0
56 stars 75 forks source link

make our dep-8 tests non-superficial #2610

Open orndorffgrant opened 1 year ago

orndorffgrant commented 1 year ago

Our dep-8/autopkgtest suite is superficial right now. We should port some of our most important feature tests to dep-8 scripts so that they can run on changes to dependencies/reverse-dependencies on launchpad. These will most likely need to be "isolation-container" or "isolation-machine" at least. See https://people.debian.org/~eriberto/README.package-tests.html

orndorffgrant commented 1 year ago

https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst

Decision: Low priority, but we could probably run our unattached commands.

renanrodrigo commented 1 year ago

@bryceharrington suggested some cases for our dep-8 tests in the Relase 30 review -

It occurs to me as well that this cache initialization behavior (esm-cache init)
would be highly appropriate for invoking in a Dep8 test, since that
could catch issues if for some reason apt's behavior changed in a newer
release or something.

and

I imagine in general a basic dep8 test ought do:
  - Retrieve a token for testing purposes
  - Attach a machine
  - Activate a subscription
  - Identify updates
  - Apply a fix
  - Check status
orndorffgrant commented 8 months ago

After we have non-superficial tests, we should make sure CI/jenkins do not accept an 8 return code from autopkgtest runs as a success

orndorffgrant commented 1 month ago

It unfortunately isn't feasible to fully test the primary functionality of pro in autopkgtests because of network limitations in the infrastructure. We have added https://github.com/canonical/ubuntu-pro-client/commit/a5d00b71c0deec98bd383aebf60d7e8f70e97ab6 though, which is a substantial improvement and will catch any breaking changes introduced by our biggest dependency: python3-apt.

We think this is enough to remove the superficial tag.