microsoft / azserialconsole

Azure Serial Console
17 stars 11 forks source link

KNOWN ISSUE: Azure Serial Console connectivity issues with certain Azure v5 series Virtual Machines #96

Closed kof-f closed 2 years ago

kof-f commented 2 years ago

Hey All,

We have become aware of an issue with our service and have created this issue to track it and update you all on the status of the fix. Please see the details below for more information.

Issue: Azure Serial Console requests to certain Azure v5 series virtual machines consistently result in the error “Sorry, the serial console was unable to connect to the VM because the service did not respond in a timely manner.”

Cause: An incompatibility has been discovered between the Azure Serial Console service and the 3rd Generation Intel® Xeon® Platinum 8370C (Ice Lake) processor. This incompatibility causes serial console connections to fail for virtual machine series which use this processor (listed below). The bi-directional Azure Serial Console is impacted by this incompatibility, however, serial logs and screenshots in boot diagnostics continue to be available for virtual machine troubleshooting.

Impacted Virtual machines: The affected virtual machine series include: Dv5 and Dsv5-series, Ddv5 and Ddsv5-series, Ev5 and Esv5-series, Edv5 and Edsv5-series and Ebsv5/Ebdsv5 series preview.

Mitigation: Resizing the virtual machine to a virtual machine series not affected (e.g. Dv4) will enable a successful Azure Serial Console connection.

Resolution status: The fix for this issue has been developed and is being validated for broad deployment. We expect complete deployment of the fix in early calendar year 2022.

samueljamkhandi-MSFT commented 2 years ago

@kof-f - could you please help us tentative month when we can expect this fix to be rolled.

kof-f commented 2 years ago

UPDATE: A fix for this issue is in deployment. Serial Console users will see relief from the error as the change rolls out. We expect the change to be broadly deployed by the end of April 2022.

ghost commented 2 years ago

Could the Standard D8ds v5 and Standard E32ds v5 instances also be affected, since we have tried really everything, but the connect to the instance just does not work? Just wanting to confirm, not that we are waiting for a fix here and the problem lies somewhere else. Both have Intel(R) Xeon(R) Platinum 8370C CPUs

samueljamkhandi-MSFT commented 2 years ago

@autonompost - AFAIK, Azure V5 series VMs are impacted.

https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-errors

image

wobblewobble commented 2 years ago

Seeing the same issue with Standard_D4s_v3

CraigWiand commented 2 years ago

UPDATE: We were able to prioritize the deployment of this fix and it has now reached >97% of Azure nodes. The majority of Serial Console users will see relief from the issue. As the update continues in deployment, we now expect it to reach >99% of nodes by end of February 2022.

ghost commented 2 years ago

@CraigWiand Thanks that fixed our problems.

CraigWiand commented 2 years ago

Issue should no longer be present for v5 VMs

samueljamkhandi-MSFT commented 2 years ago

Many thanks @CraigWiand for the confirmation.