search

Azure / azure-linux-extensions

Linux Virtual Machine Extensions for Azure
Apache License 2.0
301 stars 251 forks source link

Backup not possible from Azure Portal with waagent + Ubuntu 24.04 LTS #1922

Closed pthoelken closed 1 month ago

pthoelken commented 2 months ago

Describe the bug:

Since the update from Ubuntu 20.04 LTS to 24.04 LTS ( 20.04 LTS -> 22.04 LTS -> 23.04 -> 24.04 LTS, normal procedure) I've got daily the message from the Azure monitoring portal, that the backup was unsucessful. After checking the log files, I've got the error message from below while the agent try to create the snapshot.

Distro and WALinuxAgent details:

Log file attached

2024-05-29T05:45:42.987587Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0] Executing command: /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/handle.sh enable with environment variables: {"AZURE_GUEST_AGENT_UNINSTALL_CMD_EXIT_CODE": "NOT_RUN", "AZURE_GUEST_AGENT_EXTENSION_PATH": "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0", "AZURE_GUEST_AGENT_EXTENSION_VERSION": "1.0.9217.0", "AZURE_GUEST_AGENT_WIRE_PROTOCOL_ADDRESS": "XXX.XXX.XXX.XXX", "ConfigSequenceNumber": "637", "AZURE_GUEST_AGENT_EXTENSION_SUPPORTED_FEATURES": "[{\"Key\": \"ExtensionTelemetryPipeline\", \"Value\": \"1.0\"}]"}
2024-05-29T05:45:44.992799Z ERROR ExtHandler ExtHandler Event: name=Microsoft.Azure.RecoveryServices.VMSnapshotLinux, op=Enable, message=[ExtensionOperationError] Non-zero exit code: 1, /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/handle.sh enable
[stdout]

[stderr]
Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/Utils/WAAgentUtil.py", line 57, in <module>
    waagent = imp.load_source('waagent', agentPath)
              ^^^^^^^^^^^^^^^
AttributeError: module 'importlib' has no attribute 'load_source'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/handle.py", line 40, in <module>
    from mounts import Mounts
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/mounts.py", line 25, in <module>
    from Utils.DiskUtil import DiskUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/Utils/DiskUtil.py", line 29, in <module>
    import Utils.HandlerUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/Utils/HandlerUtil.py", line 69, in <module>
    import Utils.WAAgentUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/Utils/WAAgentUtil.py", line 64, in <module>
    waagent = imp.load_source('waagent', agentPath)
              ^^^^^^^^^^^^^^^
AttributeError: module 'importlib' has no attribute 'load_source'
Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/Utils/WAAgentUtil.py", line 57, in <module>
    waagent = imp.load_source('waagent', agentPath)
              ^^^^^^^^^^^^^^^
AttributeError: module 'importlib' has no attribute 'load_source'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/handle.py", line 40, in <module>
    from mounts import Mounts
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/mounts.py", line 25, in <module>
    from Utils.DiskUtil import DiskUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/Utils/DiskUtil.py", line 29, in <module>
    import Utils.HandlerUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/Utils/HandlerUtil.py", line 69, in <module>
    import Utils.WAAgentUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/main/Utils/WAAgentUtil.py", line 64, in <module>
    waagent = imp.load_source('waagent', agentPath)
              ^^^^^^^^^^^^^^^
AttributeError: module 'importlib' has no attribute 'load_source'
, duration=0
2024-05-29T05:45:44.994899Z INFO ExtHandler ExtHandler ProcessExtensionsGoalState completed [etag_8166643226849683953 4186 ms]

2024-05-29T05:45:52.997921Z WARNING ExtHandler ExtHandler [PERIODIC] This status is being reported by the Guest Agent since no status file was reported by extension Microsoft.Azure.RecoveryServices.VMSnapshotLinux: [ExtensionStatusError] Status file /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9217.0/status/637.status does not exist
2024-05-29T05:45:53.017470Z INFO ExtHandler ExtHandler Extension status: [('Microsoft.Azure.RecoveryServices.VMSnapshotLinux', 'transitioning'), ('Microsoft.CPlat.Core.RunCommandLinux', 'success')]
gjirm commented 2 months ago

I experience same behavior but I have clean Ubuntu 24.04 LTS (GNU/Linux 6.8.0-1007-azure x86_64) installed from:

"publisherName" = "Canonical",
"offerName" = "ubuntu-24_04-lts",
"skuName" = "server",
"imageVersion" = "latest",

WALinuxAgent details:

WALinuxAgent-2.9.1.1 running on ubuntu 24.04
Python: 3.12.3
Goal state agent: 2.11.1.4

Log file:

2024-05-30T13:15:45.331589Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0] Enable extension: [main/handle.sh enable]
2024-05-30T13:15:45.332037Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0] Executing command: /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/handle.sh enable with environment variables: {"AZURE_GUEST_AGENT_UNINSTALL_CMD_EXIT_CODE": "NOT_RUN", "AZURE_GUEST_AGENT_EXTENSION_PATH": "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0", "AZURE_GUEST_AGENT_EXTENSION_VERSION": "1.0.9218.0", "AZURE_GUEST_AGENT_WIRE_PROTOCOL_ADDRESS": "168.63.129.16", "ConfigSequenceNumber": "25", "AZURE_GUEST_AGENT_EXTENSION_SUPPORTED_FEATURES": "[{\"Key\": \"ExtensionTelemetryPipeline\", \"Value\": \"1.0\"}]"}
2024-05-30T13:15:47.339279Z ERROR ExtHandler ExtHandler Event: name=Microsoft.Azure.RecoveryServices.VMSnapshotLinux, op=Enable, message=[ExtensionOperationError] Non-zero exit code: 1, /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/handle.sh enable
[stdout]

[stderr]
Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/Utils/WAAgentUtil.py", line 57, in <module>
    waagent = imp.load_source('waagent', agentPath)
              ^^^^^^^^^^^^^^^
AttributeError: module 'importlib' has no attribute 'load_source'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/handle.py", line 40, in <module>
    from mounts import Mounts
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/mounts.py", line 25, in <module>
    from Utils.DiskUtil import DiskUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/Utils/DiskUtil.py", line 29, in <module>
    import Utils.HandlerUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/Utils/HandlerUtil.py", line 69, in <module>
    import Utils.WAAgentUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/Utils/WAAgentUtil.py", line 64, in <module>
    waagent = imp.load_source('waagent', agentPath)
              ^^^^^^^^^^^^^^^
AttributeError: module 'importlib' has no attribute 'load_source'
Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/Utils/WAAgentUtil.py", line 57, in <module>
    waagent = imp.load_source('waagent', agentPath)
              ^^^^^^^^^^^^^^^
AttributeError: module 'importlib' has no attribute 'load_source'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/handle.py", line 40, in <module>
    from mounts import Mounts
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/mounts.py", line 25, in <module>
    from Utils.DiskUtil import DiskUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/Utils/DiskUtil.py", line 29, in <module>
    import Utils.HandlerUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/Utils/HandlerUtil.py", line 69, in <module>
    import Utils.WAAgentUtil
  File "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/main/Utils/WAAgentUtil.py", line 64, in <module>
    waagent = imp.load_source('waagent', agentPath)
              ^^^^^^^^^^^^^^^
AttributeError: module 'importlib' has no attribute 'load_source'
, duration=0
2024-05-30T13:15:47.341926Z INFO ExtHandler ExtHandler ProcessExtensionsGoalState completed [etag_7315835131380991275 2104 ms]

2024-05-30T13:15:55.344457Z WARNING ExtHandler ExtHandler [PERIODIC] This status is being reported by the Guest Agent since no status file was reported by extension Microsoft.Azure.RecoveryServices.VMSnapshotLinux: [ExtensionStatusError] Status file /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9218.0/status/25.status does not exist
2024-05-30T13:15:55.367164Z INFO ExtHandler ExtHandler Extension status: [('Microsoft.Azure.RecoveryServices.VMSnapshotLinux', 'transitioning')]
2024-05-30T13:15:55.367990Z INFO ExtHandler ExtHandler Looking for existing remote access users.
2024-05-30T13:15:55.387018Z INFO ExtHandler ExtHandler [HEARTBEAT] Agent WALinuxAgent-2.11.1.4 is running as the goal state agent [DEBUG HeartbeatCounter: 0;HeartbeatId: 64C0332B-CBEA-4E84-8B14-EDFE79902E87;DroppedPackets: 5;UpdateGSErrors: 0;AutoUpdate: 1]
pthoelken commented 2 months ago

Sounds like a incompability from Azure Linux Extension to Ubuntu 24.04 @andyliuliming or @simathih may you have more information about this for us?

simathih commented 2 months ago

Yeah, Ubuntu 24 is not yet supported as per https://learn.microsoft.com/en-us/azure/azure-monitor/agents/agents-overview#linux

pthoelken commented 2 months ago

Thanks for sharing this information @simathih. Can you tell us when the support for Debian 12 and Ubuntu 24.04 LTS is available?

simathih commented 2 months ago

@lukeout can comment on timelines for Ubuntu 24 support

pthoelken commented 1 month ago

Any updates in this case? Azure offers me a Ubuntu 24.04 virtual machine without support for the azure agent client ... makes no sense for me.

simathih commented 1 month ago

adding @ankitha-p as well

ankitha-p commented 1 month ago

Hi, this is a known issue. It is fixed in VMSnapshotLinux extension version 1.0.9220.0 (https://github.com/Azure/azure-linux-extensions/pull/1920) and is currently being rolled out. It should be available in all regions by end of June.

pthoelken commented 1 month ago

@ankitha-p can you provide a link th a deb so I will install the package manually. Is there anything to do after installing or is the service re-connecting automaticly to azure portal?

pthoelken commented 1 month ago

@ankitha-p - any updates in this case? Can you provide the deb file (download link) manually? Currently I'm not able to do any backups from my machine into Azure Portal since about one month. This is not good, either this fact, that Azure offers me to install Ubuntu 24.04 (which is not supported by the agent) on my Azure virtual machines.

ankitha-p commented 1 month ago

Hi @pthoelken, the latest extension version 9220 has been rolled out to all regions. The rollout completed today. The extension version in your VM should reflect this number if a backup is initiated today. Please let us know if you don't see the latest package. We can suggest alternatives then.

pthoelken commented 1 month ago

Hi @ankitha-p how and where can I check the versions? It seems to be that the backups working now. Is the updated automcatily installed by the rollout?

ankitha-p commented 1 month ago

Please check the version after triggering a new backup job. You can check at /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-

pthoelken commented 1 month ago

Yep ... this is installed .. but not by me. Thanks for sharing this information. How was is installed? Is there an auto-update function?

ankitha-p commented 1 month ago

Once the backup job is completed can you confirm if it is successful? A new backup job checks for the latest VMSnapshotLinux extension version and installs it automatically.

pthoelken commented 1 month ago

Yep. I've checkd this in the A-backupcenter, it was successfull yesterday. Thanks :)