More apparmor fixes - Githubissues

panlinux commented 2 months ago

Why is this needed?

Further automated testing showed a few more apparmor rules to be needed. Fixes: #3079

Test Steps

Run automated tests here in GH, and the ones from jenkins.

Checklist

[ ] I have updated or added any unit tests accordingly
[ ] I have updated or added any integration tests accordingly
[ ] Changes here need to be documented, and this was done in:

Does this PR require extra reviews?

[ ] Yes
[x] No

github-actions[bot] commented 2 months ago

Jira: This PR is not related to a Jira item. (The PR title does not include a SC-#### reference)

GitHub Issues:

Fixes: #3079

Launchpad Bugs: No Launchpad bugs are fixed by this PR. (No commits have LP: #### references)

Documentation: The changes in this PR do not require documentation changes.

👍 this comment to confirm that this is correct.

panlinux commented 2 months ago

Uh, why were just 7 checks run? No behave runs?

panlinux commented 2 months ago

Ok, I made a commit message change and force-pushed, now the behave tests were triggered.

panlinux commented 2 months ago

apt-mirror is failing in jammy:

2024-05-01T18:50:24.0204512Z       2024-05-01 18:50:21,980:INFO:pycloudlib.instance:executing: sudo apt-mirror
2024-05-01T18:50:24.0205838Z       2024-05-01 18:50:22,060:ERROR:root:Error executing command: apt-mirror
2024-05-01T18:50:24.0206965Z       2024-05-01 18:50:22,060:ERROR:root:stdout: 
2024-05-01T18:50:24.0208872Z       2024-05-01 18:50:22,060:ERROR:root:stderr: Use of uninitialized value $config_line***"type"*** in string eq at /usr/bin/apt-mirror line 326, <CONFIG> line 5.
2024-05-01T18:50:24.0212245Z       Use of uninitialized value $config_line***"type"*** in string eq at /usr/bin/apt-mirror line 329, <CONFIG> line 5.
2024-05-01T18:50:24.0215155Z       Use of uninitialized value $config_line***"type"*** in string eq at /usr/bin/apt-mirror line 334, <CONFIG> line 5.
2024-05-01T18:50:24.0217517Z       Use of uninitialized value $config_line***"type"*** in pattern match (m//) at /usr/bin/apt-mirror line 337, <CONFIG> line 5.
2024-05-01T18:50:24.0224894Z       apt-mirror: invalid line in config file (5:         deb https://bearer:mAgJbEWNBSDZKVnhHb1ZYMjUtZmRrMk10Z1M4X2RsN1pvUnYzem1nQWtvY2lJeWxzOnVhLWFpcmdhcHBlZC1VRFQ4TDZGYW5IQ1hBVkJPZlVyQVFtRkh5M3l2Z2lXMQACOGlzLWNvbnRyYWN0IGNBSDZKVnhHb1ZYMjUtZmRrMk10Z1M4X2RsN1pvUnYzem1nQWtvY2lJeWxzAAIVaXMtcmVzb3VyY2UgZXNtLWluZnJhAAAGIA4-wRGa-B1AKqpe-_prx5f760UobsCwjCJs0rFBDjJD
2024-05-01T18:50:24.0228924Z        ...) at /usr/bin/apt-mirror line 350, <CONFIG> line 5.
2024-05-01T18:50:24.0231024Z       2024-05-01 18:50:22,060:WARNING:root:STEP FAILED. Collecting logs.

Has somebody seen this before? That looks like a long password, where is the rest of the line, is this a sources.list -like line? (And I hope that token is ephemeral, otherwise it just leaked)

panlinux commented 2 months ago

In xenial we have lots of failures. One is:

2024-05-01T19:12:36.7375783Z     When I apt upgrade                                                                                                                                                     # features/steps/packages.py:49
2024-05-01T19:12:36.9904988Z     Then I will see the following on stdout                                                                                                                                # features/steps/output.py:15
2024-05-01T19:12:36.9906450Z       """
2024-05-01T19:12:36.9907116Z       Reading package lists...
2024-05-01T19:12:36.9907895Z       Building dependency tree...
2024-05-01T19:12:36.9908676Z       Reading state information...
2024-05-01T19:12:36.9909479Z       Calculating upgrade...
2024-05-01T19:12:36.9910404Z       0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2024-05-01T19:12:36.9911395Z       """
2024-05-01T19:12:38.8965482Z       Assertion Failed: Expected to find exactly:
2024-05-01T19:12:38.8966480Z         Reading package lists...
2024-05-01T19:12:38.8967296Z         Building dependency tree...
2024-05-01T19:12:38.8968124Z         Reading state information...
2024-05-01T19:12:38.8968934Z         Calculating upgrade...
2024-05-01T19:12:38.8969976Z         0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2024-05-01T19:12:38.8971205Z       But got:
2024-05-01T19:12:38.8972263Z         Reading package lists...
2024-05-01T19:12:38.8973684Z         Building dependency tree...
2024-05-01T19:12:38.8974693Z         Reading state information...
2024-05-01T19:12:38.8975605Z         Calculating upgrade...
2024-05-01T19:12:38.8976473Z         The following packages have been kept back:
2024-05-01T19:12:38.8977610Z           liblxc1 lxd-client
2024-05-01T19:12:38.8978581Z         0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

panlinux commented 2 months ago

And why would this be failing in xenial? It is an LTS release (see last line):

2024-05-01T19:38:58.6482123Z     When I run shell command `env LC_CTYPE=pt_BR.UTF-8 LANGUAGE=pt_BR.UTF-8 pro security-status` as non-root                                   # features/steps/shell.py:107
2024-05-01T19:38:59.2869645Z     Then stdout contains substring                                                                                                             # features/steps/output.py:148
2024-05-01T19:38:59.2871151Z       """
2024-05-01T19:38:59.2872198Z       This machine is NOT attached to an Ubuntu Pro subscription.
2024-05-01T19:38:59.2873383Z       """
2024-05-01T19:39:00.9092664Z       Assertion Failed: Expected to find substring:
2024-05-01T19:39:00.9094456Z         This machine is NOT attached to an Ubuntu Pro subscription.
2024-05-01T19:39:00.9095961Z       But couldn't find it in:
2024-05-01T19:39:00.9096898Z         444 packages installed:
2024-05-01T19:39:00.9098219Z             441 packages from Ubuntu Main/Restricted repository
2024-05-01T19:39:00.9099536Z             3 packages no longer available for download
2024-05-01T19:39:00.9100600Z       
2024-05-01T19:39:00.9101506Z         To get more information about the packages, run
2024-05-01T19:39:00.9102700Z             pro security-status --help
2024-05-01T19:39:00.9103693Z         for a list of available options.
2024-05-01T19:39:00.9104601Z       
2024-05-01T19:39:00.9105517Z         Ubuntu Pro is not available for non-LTS releases.

panlinux commented 2 months ago

And in bionic, there is this exception when taking a snapshot of the vm:

2024-05-01T18:56:50.9938988Z     When I reboot the machine                                                                                     # features/steps/machines.py:235
2024-05-01T18:58:32.3070087Z     When I take a snapshot of the machine                                                                         # features/steps/machines.py:142
2024-05-01T19:01:13.8347010Z       Captured stdout:
2024-05-01T19:01:13.8348419Z       HOOK-ERROR in after_step: SSHException: 
2024-05-01T19:01:13.8354028Z         File "/home/runner/work/ubuntu-pro-client/ubuntu-pro-client/.tox/behave/lib/python3.10/site-packages/behave/runner.py", line 545, in run_hook
2024-05-01T19:01:13.8356829Z           self.hooks[name](context, *args)
2024-05-01T19:01:13.8358490Z         File "features/environment.py", line 539, in after_step
2024-05-01T19:01:13.8360362Z           apparmor_logs = _get_relevant_apparmor_logs(context)
2024-05-01T19:01:13.8362433Z         File "features/environment.py", line 525, in _get_relevant_apparmor_logs
2024-05-01T19:01:13.8364485Z           sut.instance.pull_file("/var/log/syslog", syslog_dest)
2024-05-01T19:01:13.8367694Z         File "/home/runner/work/ubuntu-pro-client/ubuntu-pro-client/.tox/behave/lib/python3.10/site-packages/pycloudlib/instance.py", line 314, in pull_file
2024-05-01T19:01:13.8370277Z           sftp = self._sftp_connect()
2024-05-01T19:01:13.8373203Z         File "/home/runner/work/ubuntu-pro-client/ubuntu-pro-client/.tox/behave/lib/python3.10/site-packages/pycloudlib/instance.py", line 472, in _sftp_connect
2024-05-01T19:01:13.8375860Z           client = self._ssh_connect()
2024-05-01T19:01:13.8379048Z         File "/home/runner/work/ubuntu-pro-client/ubuntu-pro-client/.tox/behave/lib/python3.10/site-packages/pycloudlib/instance.py", line 457, in _ssh_connect
2024-05-01T19:01:13.8381639Z           raise SSHException from e

I don't know what the exception actually was. Failed to connect? Maybe the VM didn't reboot properly?

orndorffgrant commented 2 months ago

apt-mirror is failing in jammy:

This one should be fixed by https://github.com/canonical/ubuntu-pro-client/pull/3092

orndorffgrant commented 2 months ago

And in bionic, there is this exception when taking a snapshot of the vm:

This looks like a race condition between the vm booting back up after a snapshot and the apparmor check trying to pull syslog.

I think I'll just modify the apparmor check to print a warning on an SSH exception and continue

panlinux commented 2 months ago

And in bionic, there is this exception when taking a snapshot of the vm:

I think I'll just modify the apparmor check to print a warning on an SSH exception and continue

What do you mean "apparmor check"? Is this related to apparmor?

panlinux commented 2 months ago

apt-mirror is failing in jammy:

This one should be fixed by #3092

Ok, I can rebase once that lands.

orndorffgrant commented 2 months ago

And in bionic, there is this exception when taking a snapshot of the vm:

I think I'll just modify the apparmor check to print a warning on an SSH exception and continue

What do you mean "apparmor check"? Is this related to apparmor?

Only related to the after_step check for apparmor denial logs. From the stacktrace:

File "features/environment.py", line 539, in after_step
  apparmor_logs = _get_relevant_apparmor_logs(context)
File "features/environment.py", line 525, in _get_relevant_apparmor_logs
  sut.instance.pull_file("/var/log/syslog", syslog_dest)

We're trying to pull_file but the snapshot step that this is executing after rebooted the VM. It seems SSH on the VM wasn't quite ready for new connections when this after_step code runs

panlinux commented 2 months ago

And in bionic, there is this exception when taking a snapshot of the vm:

I think I'll just modify the apparmor check to print a warning on an SSH exception and continue

What do you mean "apparmor check"? Is this related to apparmor?

Only related to the after_step check for apparmor denial logs. From the stacktrace:
File "features/environment.py", line 539, in after_step
  apparmor_logs = _get_relevant_apparmor_logs(context)
File "features/environment.py", line 525, in _get_relevant_apparmor_logs
  sut.instance.pull_file("/var/log/syslog", syslog_dest)
We're trying to pull_file but the snapshot step that this is executing after rebooted the VM. It seems SSH on the VM wasn't quite ready for new connections when this after_step code runs

Ah, right, we unconditionally get the logs everytime.

orndorffgrant commented 2 months ago

And why would this be failing in xenial? It is an LTS release (see last line):

There will be an SRU of distro-info-data to xenial to fix this one

lucasmoura commented 2 months ago

@panlinux I have tested some Xenial tests using distro-info-data from proposed and I ran a subset of the affected Xenial tests and I was only able to identify one valid error, which I have committed a fix for

panlinux commented 2 months ago

Ok, now we only have xenial failures. Quite a few, though:

Failing scenarios:
  features/api_fix_execute.feature:1101  Fix execute API command on a Xenial machine -- @1.1 ubuntu release details
  features/api_fix_plan.feature:1788  Fix command on an unattached machine -- @1.1 ubuntu release details
  features/apt_messages.feature:133  APT Hook advertises esm-infra on upgrade -- @1.1 ubuntu release
  features/apt_messages.feature:646  APT News -- @1.1 ubuntu release
  features/apt_messages.feature:1052  APT news selectors -- @1.1 ubuntu release
  features/fix.feature:592  Fix command on an unattached machine -- @1.1 ubuntu release details
  features/i18n.feature:85  Translation doesn't error when python thinks it's ascii only -- @1.1 ubuntu release
  features/motd_messages.feature:141  Contract Expiration Messages -- @1.1 ubuntu release
  features/security_status.feature:88  Run security status with JSON/YAML format -- @1.1 ubuntu release
  features/security_status.feature:111  Run security status in an Ubuntu machine

panlinux commented 2 months ago

apt_messages.feature:133: missing update?

Assertion Failed: Expected to match regexp:
  Reading package lists...
  Building dependency tree...
  Reading state information...
  Calculating upgrade...
  The following security updates require Ubuntu Pro with 'esm-infra' enabled:
    ([-+.\w\s]*)
  Learn more about Ubuntu Pro for 16\.04 at https:\/\/ubuntu\.com\/16-04
  0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded\.
But got:
  Reading package lists...
  Building dependency tree...
  Reading state information...
  Calculating upgrade...
  0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

panlinux commented 2 months ago

apt_messages.feature:646 unexpected update?

Assertion Failed: Expected to find exactly:
  Reading package lists...
  Building dependency tree...
  Reading state information...
  Calculating upgrade...
  #
  # one
  #
  0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
But got:
  Reading package lists...
  Building dependency tree...
  Reading state information...
  Calculating upgrade...
  #
  # one
  #
  The following packages have been kept back:
    liblxc1 lxd-client
  0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

panlinux commented 2 months ago

apt_messages.feature:1052 also unexpected update?

Assertion Failed: Expected to find exactly:
  Reading package lists...
  Building dependency tree...
  Reading state information...
  Calculating upgrade...
  0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
But got:
  Reading package lists...
  Building dependency tree...
  Reading state information...
  Calculating upgrade...
  The following packages have been kept back:
    liblxc1 lxd-client
  0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

panlinux commented 2 months ago

fix.feature:592 seems to be missing the warning text about running as non-root:

panlinux commented 2 months ago

i18n.feature:85 is about xenial not being recognized as an lts:

Assertion Failed: Expected to find substring:
  This machine is NOT attached to an Ubuntu Pro subscription.
But couldn't find it in:
  444 packages installed:
      441 packages from Ubuntu Main/Restricted repository
      3 packages no longer available for download

  To get more information about the packages, run
      pro security-status --help
  for a list of available options.

  Ubuntu Pro is not available for non-LTS releases.

The fix for that is currently in xenial-proposed: https://launchpad.net/ubuntu/+source/distro-info-data/0.28ubuntu0.19

panlinux commented 2 months ago

motd_messages.feature:141 output is different, contains more text than expected, unsure what is going on

Assertion Failed: Expected to match regexp:
  [\w\d.]+

  \*Your Ubuntu Pro subscription has EXPIRED\*
  \d+ additional security update(s)? require(s)? Ubuntu Pro with 'esm-infra' enabled.
  Renew your subscription at https:\/\/ubuntu.com\/pro\/dashboard
But got:
  Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 5.15.0-1061-azure x86_64)

   * Documentation:  https://help.ubuntu.com
   * Management:     https://landscape.canonical.com    
   * Support:        https://ubuntu.com/advantage

  UA Apps: Extended Security Maintenance (ESM) is enabled.

  196 updates can be applied immediately.
  188 of these updates are UA Infra: ESM security updates.
  2 of these updates are UA Apps: ESM security updates. 
  3 of these updates are standard security updates.
  To see these additional updates run: apt list --upgradable 

  *Your Ubuntu Pro subscription has EXPIRED*
  Renew your subscription at https://ubuntu.com/pro/dashboard

  New release '18.04.6 LTS' available.
  Run 'do-release-upgrade' to upgrade to it.

orndorffgrant commented 2 months ago

@panlinux many of these are effects of the distro-info-data bug - I'll run all the failed tests with the updated distro-info-data to look for unrelated failures (e.g. the liblxc held back one will need a separate fix)

panlinux commented 2 months ago

security_status.feature:88 change in updates? apport is not in the list: (I wrapped the lines below)

Assertion Failed: Expected to match regexp:
  "package": "apport"
But got: 
  ***"_schema_version": "0.1", "livepatch": ***"fixed_cves": []***, 
  "packages": [***"download_size": 29410, "origin": "security.ubuntu.com",
    "package": "libapparmor1", "service_name": "standard-security", "status": "upgrade_available", "version": "2.10.95-0ubuntu2.12"***, ***"download_size": 31608, "origin": "security.ubuntu.com", 
    "package": "libapparmor-perl", "service_name": "standard-security", "status": "upgrade_available", "version": "2.10.95-0ubuntu2.12"***, ***"download_size": 450598, "origin": "security.ubuntu.com",
    "package": "apparmor", "service_name": "standard-security", "status": "upgrade_available", "version": "2.10.95-0ubuntu2.12"***],
  "summary": ***"num_esm_apps_packages": 0, "num_esm_apps_updates": 0, "num_esm_infra_packages": 0, "num_esm_infra_updates": 0, "num_installed_packages": 464, "num_main_packages": 459, "num_multiverse_packages": 0, "num_restricted_packages": 0, "num_standard_security_updates": 3, "num_third_party_packages": 0, "num_universe_packages": 2, "num_unknown_packages": 3, "reboot_required": "no", "ua": ***"attached": false, "enabled_services": [], "entitled_services": []***

panlinux commented 2 months ago

security_status.feature:111 should also be fixed by the distro-info-update I suspect:

Assertion Failed: Expected to match regexp:
  \d+ packages installed:
   +\d+ package[s]? from Ubuntu Main/Restricted repository
   +\d+ package[s]? from Ubuntu Universe/Multiverse repository
   +\d+ package[s]? from a third party 
   +\d+ package[s]? no longer available for download 

  To get more information about the packages, run
      pro security-status --help
  for a list of available options\.  

  This machine is NOT receiving security patches because the LTS period has ended
  and esm-infra is not enabled.
  This machine is NOT attached to an Ubuntu Pro subscription.

  Ubuntu Pro with 'esm-infra' enabled provides security updates for
  Main/Restricted packages until 2026\. There (is|are) \d+ pending security update[s]?\.

  Ubuntu Pro with 'esm-apps' enabled provides security updates for
  Universe/Multiverse packages until 2026\. There (is|are) \d+ pending security update[s]?\.

  Try Ubuntu Pro with a free personal subscription on up to 5 machines.
  Learn more at https://ubuntu.com/pro
But got:
  466 packages installed:
      459 packages from Ubuntu Main/Restricted repository
      2 packages from Ubuntu Universe/Multiverse repository
      1 package from a third party
      4 packages no longer available for download

  To get more information about the packages, run
      pro security-status --help
  for a list of available options.

  Ubuntu Pro is not available for non-LTS releases.

panlinux commented 2 months ago

api_fix_plan.feature:1788 and api_fix_execute.feature:1101 I'm not sure, the output is a bit hard to read. Looks like the vulnerabilities were not fixed. Perhaps because xenial is not recognized as an LTS? Or a change in the available updates.

panlinux commented 2 months ago

The distro-info-data update is published in xenial-updates, retriggering tests.

panlinux commented 2 months ago

Well, now he have two problems.

a) new failures because distro-info-data is an avaliable update that some tests didn't expect; b) phasing: are we ignoring phasing, or subject to it?

Regarding (a):

Assertion Failed: Expected to find exactly:
  Reading package lists...
  Building dependency tree...
  Reading state information...
  Calculating upgrade...
  #
  # one
  #
  0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
But got:
  Reading package lists...
  Building dependency tree...
  Reading state information...
  Calculating upgrade...
  #
  # one
  #
  The following packages have been kept back:
    liblxc1 lxd-client
  The following packages will be upgraded:
    distro-info-data
  1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

panlinux commented 2 months ago

I'm also still seeing Ubuntu Pro is not available for non-LTS releases. in some tests, do we need a new image with the updated distro-info-data included? I suppose we don't start fresh with a dist-upgrade exactly because some tests require available updates?

orndorffgrant commented 2 months ago

I've run all the tests with the setup modified to ensure the new distro-info-data was installed and the only remaining failure is: features/api_fix_plan.feature:1788 - I haven't triaged the failure yet

@panlinux should we modify our Depends to require the new version of distro-info-data?

panlinux commented 2 months ago

@panlinux should we modify our Depends to require the new version of distro-info-data?

No, better not, packages rarely have such a versioned dependency when an update is available.

orndorffgrant commented 2 months ago

Okay turns out even features/api_fix_plan.feature:1788 was a false positive. I just forgot to have it run apt update after installing the new distro-info-data.

So I think this is good to merge unless you have any last reservations @panlinux ?

Merging it will also trigger CI to run again over on #3052

canonical / ubuntu-pro-client

More apparmor fixes #3096

Why is this needed?

Test Steps

Checklist

Does this PR require extra reviews?