Script stops executing when running apt-get using Microsoft.Compute/virtualMachines/runCommands@2021-07-01

[PeterBrymer]

Bicep version Bicep CLI version 0.12.40 (41892bd0fb)

Describe the bug When using apt-get update, upgrade & install commands when using the Microsoft.Compute/virtualMachines/runCommands@2021-07-01 on the following vm specs the script stop executing after the first apt-get command. Operating system: Linux (ubuntu 22.04) Publisher: Canonical Offer: 0001-com-ubuntu-server-jammy Plan: 22_04-lts

To Reproduce

1. Have a vm like above.

2. Create a bicep file with the following content:

   param Location string = resourceGroup().location
   var vmName = 'nameOfVM'
   var name = 'test'
   var scriptContentupgradeVM = loadTextContent('vm-upgrade.sh', 'utf-8')
   resource upgradeVM 'Microsoft.Compute/virtualMachines/runCommands@2021-07-01' = {
   name: take('${vmName}/${name}-upgradeVM', 64)
   location: Location
   properties: {
   asyncExecution: false
   parameters: []
   protectedParameters: []
   source: {
     script: scriptContentupgradeVM
   }
   timeoutInSeconds: 120
   }
   }

3. Have a script with the following content saved with the name vm-upgrade.sh in same folder as the bicep file so the bicep file can get the content from it: #!/bin/bash export DEBIAN_FRONTEND=noninteractive sudo echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections sudo sed -i 's/#$nrconf{restart} = '"'"'i'"'"';/$nrconf{restart} = '"'"'a'"'"';/g' /etc/needrestart/needrestart.conf echo 'hello1' >> /tmp/hellofile.txt sudo apt-get update echo 'hello2' >> /tmp/hellofile.txt sudo apt-get upgrade -y echo 'hello3' >> /tmp/hellofile.txt

4. run: az deployment group create --resource-group rgName --template-file bicepFileName

5. Then logon to the vm and run cat /tmp/hellofile.txt and you will se that only the hello1 echo has been printed to the log and no other one after that. Then run: ls -l /var/cache/apt/pkgcache.bin | cut -d' ' -f6,7,8 and you will se that the apt package cache has been update so the sudo apt-get update command has been run but then after that the scripts just stops executing. Example: azureuser@vm-cozyfish:~$ cat /tmp/hellofile.txt hello1 azureuser@vm-cozyfish:~$ ls -l /var/cache/apt/pkgcache.bin | cut -d' ' -f6,7,8 Nov 11 16:04 azureuser@vm-cozyfish:~$ date Fri Nov 11 16:05:49 UTC 2022

Additional context

I have also tried running the script by it self in the vm:s GUI RunCommand box and when i do that it´s working as expected and all hello echo's has been printed to the tmp file. So my conclusion is that it´s something wrong with the bicep module.

[alex-frankel]

There may be something strange happening when we load the script into the template. Can you share the compiled ARM template by running az bicep build?

[PeterBrymer]

There may be something strange happening when we load the script into the template. Can you share the compiled ARM template by running az bicep build?

Yes @alex-frankel, here it is:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "0.12.40.16777",
      "templateHash": "11662099839264801839"
    }
  },
  "parameters": {
    "Location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]"
    }
  },
  "variables": {
    "vmName": "vm-cozyfish",
    "name": "test1",
    "scriptContentupgradeVM": "#!/bin/bash\nexport DEBIAN_FRONTEND=noninteractive\nsudo echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections\nsudo sed -i 's/#$nrconf{restart} = '\"'\"'i'\"'\"';/$nrconf{restart} = '\"'\"'a'\"'\"';/g' /etc/needrestart/needrestart.conf\necho 'hello1' >> /tmp/hellofile.txt\nsudo apt-get update\necho 'hello2' >> /tmp/hellofile.txt\nsudo apt-get upgrade -y\necho 'hello3' >> /tmp/hellofile.txt\n"
  },
  "resources": [
    {
      "type": "Microsoft.Compute/virtualMachines/runCommands",
      "apiVersion": "2021-07-01",
      "name": "[take(format('{0}/{1}-upgradeVM', variables('vmName'), variables('name')), 64)]",
      "location": "[parameters('Location')]",
      "properties": {
        "asyncExecution": false,
        "parameters": [],
        "protectedParameters": [],
        "source": {
          "script": "[variables('scriptContentupgradeVM')]"
        },
        "timeoutInSeconds": 120
      }
    }
  ]
}

[alex-frankel]

@miqm / @anthony-c-martin -- any idea what might be happening here?

[miqm]

Seems ok to me. If the script starts executing then with bicep should be fine. Did you try to run on a different ubuntu distribution? I'd also suggest opening support ticket. Also, I remember a case we had that apt update was blocked due to autopatch addon executing at this same time and locking the package manager. Perhaps this is the case you have.

Also, there're properties to pass a storage blob uri that will be used to save script output from stdout and stderr - use them to get why you script stops running.

[PeterBrymer]

@miqm Yeah I will try it out on a different ubuntu dist and se if it works there. Where should i open an support ticket? In Azure?

I´m pretty sure it´s not an autopatch addon block as u mention because when i run the same command on the vm right before i start the bicep run it works. I have also try to added: -oDPkg::Lock::Timeout=120 at the apt-get commands and still the script stops executing after the apt-update run.

Regarding the properties to pass storageblob uri i have found an easier solution to that. The script logs are stored here together with the script in the vm: /var/lib/waagent/run-command-handler/download/RUNNAME/0/ and ls -l here looks like this:

root@vm-cozyfish:/var/lib/waagent/run-command-handler/download/test3-upgradeVM/0# ls -l
total 8
-r-x------ 1 root root 459 Nov 15 10:40 script.sh
-rw------- 1 root root   0 Nov 15 10:40 stderr
-rw------- 1 root root 479 Nov 15 10:40 stdout

when i cat the stderr file it´s empty so its not an error that happens and the cat of stdout file looks like this:

root@vm-cozyfish:/var/lib/waagent/run-command-handler/download/test3-upgradeVM/0# cat stdout
Hit:1 http://azure.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu jammy-updates InRelease
Get:3 http://azure.archive.ubuntu.com/ubuntu jammy-backports InRelease [99.8 kB]
Hit:4 http://azure.archive.ubuntu.com/ubuntu jammy-security InRelease
Hit:5 https://packages.microsoft.com/repos/azure-cli jammy InRelease
Hit:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
Fetched 99.8 kB in 1s (125 kB/s)
Reading package lists...
root@vm-cozyfish:/var/lib/waagent/run-command-handler/download/test3-upgradeVM/0#

But if i run the script.sh script inside from here it´s working as expected and both apt-get update and upgrade and my echos happens. My guess is that something goes wrong when the script run-command-handler handle the script.

[miqm]

If the script uploaded to the vm is fine then it has been loaded correctly by bicep and ARM.

Yes, open it in Azure through portal - > on the lefthand side menu choose help and support and open a ticket there. You must have a support plan though. Or if your subscription is from a CSP - then concact them - they should assist you or open a ticket to Microsoft if necessary.

[PeterBrymer]

Okey, so the problem probably is in the virtual machine then. Just for the record, i tried to do the same with a ubuntu 22.10 and it was the same as with the 22.04LTS.

We have a CSP so I will contact them and take it from there! Thanks for the answer and the guidance :)

[miqm]

Awesome! I'll close the ticket then, but if you find out that problem is in bicep - please comment here and we'll reopen.