Update management is not running for VMs of recently deployed SREs (except Guacamole)

[edwardchalstrey1]

:white_check_mark: Checklist

• [x] I have searched open and closed issues for duplicates.
• [x] This is a problem observed when managing a Data Safe Haven.
• [x] I can reproduce this with the latest version.
• [x] I have read through the documentation.
• [x] This isn't an open-ended question (open a discussion if it is).

:computer: System information

• Data Safe Haven version: 4.0.2
• Operating system details: macOS Monterey

:cactus: Powershell module versions

2023-02-24 09:58:58 [SUCCESS]: [✔] Powershell version: 7.2.6
2023-02-24 09:58:58 [SUCCESS]: [✔] Poshstache module version: 0.1.10
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.KeyVault module version: 4.6.0
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.Accounts module version: 2.9.0
2023-02-24 09:58:58 [SUCCESS]: [✔] Microsoft.Graph.Identity.DirectoryManagement module version: 1.10.0
2023-02-24 09:58:58 [SUCCESS]: [✔] Microsoft.Graph.Authentication module version: 1.17.0
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.Storage module version: 4.7.0
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.Network module version: 4.18.0
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.PrivateDns module version: 1.0.3
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.RecoveryServices module version: 5.4.1
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.Compute module version: 4.29.0
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.Resources module version: 6.0.1
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.Automation module version: 1.7.3
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.Dns module version: 1.1.2
2023-02-24 09:58:58 [SUCCESS]: [✔] Powershell-Yaml module version: 0.4.2
2023-02-24 09:58:58 [SUCCESS]: [✔] Microsoft.Graph.Applications module version: 1.17.0
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.MonitoringSolutions module version: 0.1.0
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.OperationalInsights module version: 3.1.0
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.Monitor module version: 3.0.1
2023-02-24 09:58:58 [SUCCESS]: [✔] Az.DataProtection module version: 0.4.0

:no_entry_sign: Describe the problem

For recently deployed SREs (e.g. dsgkeep) the Update management is only working for Guacamole, but not the other VMs, including the compute VM. This has resulted in packages not being updated, for example, it caused the problem that resulted in #1401

This is strange because it looks like the VM is connected to loganalytics:

:steam_locomotive: Workarounds or solutions

[jemrobinson]

Can you change the scheduled time for sre-sbox2-linux-updates from the current value (every 7 days at starting next Tuesday at 02:02) to every 1 hours starting at 03/03/2023 15:00? This way it will run in 15 minutes time and we can see what it runs over.

[edwardchalstrey1]

hmm, didn't seem to run, will investigate - oh, wrong date, well it should run at 4pm now

[edwardchalstrey1]

Weird, said it was going to run at 4 but didn't

[jemrobinson]

Maybe you just needed to wait for it to finish - seems to have run now.

if you click on this you can see which VMs it ran over and it seems to be all 5 VMs in the SRE without missing any.

so it's possible that the problem is that the query preview is not the same as what actually gets run? We can test this by changing another schedule that has been failing in the past.

[edwardchalstrey1]

For dsggw for example the latest run looks like this - so I think you may be right that sbox2 isn't a good one to test this on:

[jemrobinson]

OK, so can you update the timing on this one so it will run tonight? Just need to change start date from 28/02/2023 to 04/03/2023.

I think this might work since looking here (https://learn.microsoft.com/en-us/azure/automation/troubleshoot/update-management?WT.mc_id=Portal-Microsoft_Azure_Automation#nologs) I can see that

• required updates are listed for SRE-DSGGW-160-SRD-20-04-2022112900 from the Update Management view of the VM (see image attached below)
• the VM is visible from the Update Management view of the Azure Automation account
• it shows as "Non-compliant" under compliance (ie. it has been assessed as needing some updates)

in this particular case, it looks like the problem is that the update job simply hasn't run recently.

[jemrobinson]

Here's a VM that I think is not working and it seems to be due to an issue with the update agent.

[edwardchalstrey1]

Looks like Hybrid runbook worker is ok on this VM - I'll look at the Steps to fix Multihoming which we already identified earlier as having the extra log analytics workspace , but maybe we just need to reinstall "OMS-Agent-for-Linux" after you deleted the extra log analytics workspace @jemrobinson

I guess we expect the internet connectivity check to fail here as it's tier 3

[jemrobinson]

@edwardchalstrey1 Did fixing multihoming help? If not, did you try reinstalling the OMS agent?

[JimMadge]

Have encountered the same bug. A VM reports being connected to the Log Analytics Workspace, and Automation Account is connected to the Log Analytics Workspace. However, the automation account does not apply updates.

After troubleshooting as Ed did above, Multihoming showed as failed again. Machine should not be multihomed (connected to >1 Log Analytics Workspace).

[JimMadge]

Possible workaround is check for this and redeploy

[craddm]

An additional point is adding an SRD manually using Add_Single_SRD.ps1 doesn't enable automatic updates or install the Oms agent, so needs to be documented that Setup_SRE_Monitoring.ps1 should also be run after adding an SRD

[craddm]

On a newly deployed SRD, there are what seem to be vestigial files for omsagent in /var/opt/microsoft/omsagent/.

This id is the spurious DefaultWorkspace ID, and it is from there that the multihoming issue arises. There is already some kind of record of a workspace showing there, even though the portal shows that there are no extensions installed etc. It is possible, from expecting the logs of this omsagent, that this is something MS have running during the deployment of the VM, which never completes because there is no internet access

So I'm wondering if this is supposed to be deleted once the setup process is complete, but never is.

[jemrobinson]

Great detective work @craddm! Could be a problem that occurs during the image building process? Might be worth adding something to the deployment-time cloud-init that deletes the /var/opt/microsoft/omsagent/ directory and seeing if that helps?

[JimMadge]

Perhaps the agent gets installed (or there is an attempt) during the build of the SRD image? Ensuring those files are deleted, if they exist, in cloud-init could work as long as that happens before the agent gets installed?

[craddm]

Can confirm that the agent is there during the SRD image build

[JimMadge]

Great, I think the obvious thing to try is a step at the end of the build to rm -rf all of that.