Serial console is not working after customer enabled storage account firewall

[windswordsgd]

Serial console is not working after customer enabled storage account firewall. The issue can be reproduced. Serial console or Storage Account firewall Azure document do not mention about this limitation.

Is it by design that storage account firewall is not supported by serial console? Is there any plan for this to be ready in future

[cloudbooster]

Thanks for the feedback, this currently by design , we are looking to see if we can work around. Docs will get updated shortly.

[amitchat]

Currently there is no workaround but we will try to fix this issue in future.

[tiny-dancer]

we will try to fix this issue in future.

@amitchat, with the GA release is there any update on this being fixed in the future?

[asinn826]

@tiny-dancer We have this in our roadmap - we've received lots of feedback around this limitation so we're working on ways to resolve this. Thanks for the feedback!

[asinn826]

Hi folks - lots of feedback from other places around this, so using this GitHub issue as a place to communicate some status for interested people (I will be pointing people external to Microsoft who are blocked by boot diagnostics to this issue).

Ultimately we want to remove our dependency on the boot diagnostics storage account (https://docs.microsoft.com/en-us/azure/virtual-machines/troubleshooting/serial-console-linux#prerequisites), but this will require rearchitecting a lot of the service. We are currently going through the design phase, but we anticipate this will be a months-long chunk of work. Since we're also a GA service now, we can't take any risks on rolling out a change this big and breaking people, so that's a factor in our decision-making process :).

Let me know if you're reading this and have any questions or concerns! I'll be posting status updates as they happen.

[tiny-dancer]

Happy to wait (and additional risk) for the re-architecture of removing the storage account all together. 👍

Also, willing to wait longer to allow for a more “seamless” migration

[asinn826]

Quick update on this if people are interested...we have identified a potential new architecture that might work for us and we're now working with the teams required to make this happen. Still no eta yet but this is something we're actively looking at every day.

[asinn826]

Updating again...we're in the design phase now, we have a bit of consensus with the teams we'll be working with as to what our new architecture will look like. Now figuring out more design details and planning which changes will go where in the service. This is a pretty big overhaul of our service so we need to be deliberate in design and ultimately rollout.

[asinn826]

So, updating again... After a ton of discussion with various teams involved in our initial design, things are not moving quite as fast as we would like. We're investigating other options but this is a bit of a setback :(. I know it might not look like we're doing much on this since nothing that we've done so far is externally visible, but we're painfully aware of this limitation caused by the boot diagnostics dependency and we do want to resolve it.

[marlonsingleton]

So, updating again... After a ton of discussion with various teams involved in our initial design, things are not moving quite as fast as we would like. We're investigating other options but this is a bit of a setback :(. I know it might not look like we're doing much on this since nothing that we've done so far is externally visible, but we're painfully aware of this limitation caused by the boot diagnostics dependency and we do want to resolve it.

Thanks for the update.

[I-Am-ChuckJ]

In the meantime, Do you have any sort of workaround example, Maybe an azure function to turn off the firewall and turn it back on in 4 hrs.

[asinn826]

Depending on how you're using storage accounts, you could create a separate dedicated storage account for boot diagnostics and leave that without a firewall. A dedicated boot diagnostics storage account will only contain 3 pieces of info, all of which require proper authentication to access:

1. Serial log
2. Screenshot image
3. Serial Console connection data

That would be my suggestion for now - keep the boot diagnostics storage account separate from your other storage accounts, which minimizes the amount of data that isn't stored behind a firewall.

The Azure function suggestion is a good thought. You might be able to build some automation around the Azure CLI commands: https://docs.microsoft.com/en-us/cli/azure/storage/account/network-rule?view=azure-cli-latest

[asinn826]

We've been having lots of conversations about this issue internally and this issue is due for an update (lots of people are referring to this issue in the conversations I've been involved in).

At this point in time we are comfortable sharing that we are hoping to have Serial Console compatible with storage account firewalls by Q4 2020. A more granular eta will emerge as we get closer to the date, and the eta is subject to the usual engineering caveats regarding unknowns.

Hope this helps some people with their planning.

[dcbrown16]

Thanks @asinn826! We just had another case in advisory support yesterday that this info will be relevant for.

[JQUINONES82]

Thanks @asinn826!

[kof-f]

Hey everyone, we are long overdue for an update on this issue. We are still working hard on rectifying this, however, the Q4 2020 ETA has slipped. We don't currently have an ETA we are confident sharing externally - moving forward, I will be keeping this audience updated with our progress. Expect another update from me in a few weeks.

[kof-f]

Long overdue for an update here:

We are currently working on a preview of a short-term solution that would allow Serial Console service IP addresses to be used as firewall exclusions on custom boot diagnostics storage accounts. This preview will be made available to customers via a feature flag that enables a secure version of the Azure portal with new logic in the Serial Console service. As this workaround uses public IPs, it does not support Private Link, which we know is a requirement for some customers. We are aiming to have the preview available by Q4 2021. We don't have an ETA for when it will be available publicly yet.

We will be releasing information on how customers can register for the private preview in a few weeks. Thank you all for your patience.

[jasweet]

How about enabling this to work with a managed storage account?

Can you make it work with the "Allow trusted Microsoft services to access this storage account" toggle?

https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-linux Note: Serial Console is currently incompatible with a managed boot diagnostics storage account. To use Serial Console, ensure that you are using a custom storage account.

[kof-f]

How about enabling this to work with a managed storage account?

Can you make it work with the "Allow trusted Microsoft services to access this storage account" toggle?

https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-linux Note: Serial Console is currently incompatible with a managed boot diagnostics storage account. To use Serial Console, ensure that you are using a custom storage account.

Hey @jasweet sorry for the late response, that is also something we are currently working on.

[kof-f]

Hey everyone, we now have a private preview of the storage account firewall functionality working. If this is a blocker for your team/organization and you would like access to the private preview, please send an email to: azserialhelp@microsoft.com for access.

[pcarver]

I’m not sure if doing “Reply All” to this email will work, but I tried replying specifically to the email address mentioned below and received an auto-reply saying that said:

The group azsericon only accepts messages from people in its organization or on its allowed senders list, and your email address isn't on the list.

From: Kofi Forson @.> Sent: Wednesday, July 21, 2021 16:11 To: microsoft/azserialconsole @.> Cc: CARVER, PAUL @.>; Manual @.> Subject: Re: [microsoft/azserialconsole] Serial console is not working after customer enabled storage account firewall (#48)

Hey everyone, we now have a private preview of this functionality working. If this is a blocker for your team/organization and you would like access to the private preview, please send an email to: @.**@.> for access.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/microsoft/azserialconsole/issues/48*issuecomment-884467008__;Iw!!BhdT!2g6Ivv-zMsJI3jvpAIFTiMW7r82ElAhXzaV6UatZFtpr-wpXrOJ-NzXZzVtX$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AA4MRDZIS4AAVZESPRV3OODTY4SVPANCNFSM4E4SMEFA__;!!BhdT!2g6Ivv-zMsJI3jvpAIFTiMW7r82ElAhXzaV6UatZFtpr-wpXrOJ-NwHYlllN$.

[kof-f]

Updated the email to: azserialhelp@microsoft.com - that email should work. Thank you for pointing that out @pcarver

[noor2122]

Do we have any new update on this?

[pcarver]

I don’t think the workaround worked. As far as I know it is still necessary to disable the storage account firewall in order to use the serial console.

From: Noor Mohammad @.> Sent: Thursday, September 23, 2021 13:02 To: microsoft/azserialconsole @.> Cc: CARVER, PAUL @.>; Mention @.> Subject: Re: [microsoft/azserialconsole] Serial console is not working after customer enabled storage account firewall (#48)

Do we have any new update on this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/microsoft/azserialconsole/issues/48*issuecomment-925994162__;Iw!!BhdT!xv71iMGmdWzYsDJ_Yzjl1HhbfljyoH2UW4fBNc2mXy5zZ5efRdQvPkXFqXu4$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AA4MRD3TEQQOANCYXKBB3XLUDNMQFANCNFSM4E4SMEFA__;!!BhdT!xv71iMGmdWzYsDJ_Yzjl1HhbfljyoH2UW4fBNc2mXy5zZ5efRdQvPvVCU1JS$. Triage notifications on the go with GitHub Mobile for iOShttps://urldefense.com/v3/__https:/apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!BhdT!xv71iMGmdWzYsDJ_Yzjl1HhbfljyoH2UW4fBNc2mXy5zZ5efRdQvPtHZzMeo$ or Androidhttps://urldefense.com/v3/__https:/play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!BhdT!xv71iMGmdWzYsDJ_Yzjl1HhbfljyoH2UW4fBNc2mXy5zZ5efRdQvPuN2bCRi$.

[kof-f]

Hey everyone, we now have a private preview of the storage account firewall functionality working. If this is a blocker for your team/organization and you would like access to the private preview, please send an email to: azserialhelp@microsoft.com for access.

Hey All,

As part of our effort to better the Serial console experience, on Monday, Jan 24th, we will begin our transition to support service tags. During this transition, the existing documentation any private preview customers may have received may no longer work and the Serial console connection will again be blocked while the storage account firewall is enabled.

The transition should take about 2 weeks to complete for private preview customers. If your organization is currently using the private preview and would like a mitigation during the transition period, please send an email using the same email address you used to request access to the preview.

Thank you for your continued interest in Serial console.

[ByteBrewerJB]

I have send an email to azserialhelp but I got no response? Is someone looking to that mailbox?

[kof-f]

I have send an email to azserialhelp but I got no response? Is someone looking to that mailbox?

We are monitoring that mailbox, can you send another email if you still haven't received a response please?

[kof-f]

Hey everyone, we now have a private preview of the storage account firewall functionality working. If this is a blocker for your team/organization and you would like access to the private preview, please send an email to: azserialhelp@microsoft.com for access.

Hey All,

As part of our effort to better the Serial console experience, on Monday, Jan 24th, we will begin our transition to support service tags. During this transition, the existing documentation any private preview customers may have received may no longer work and the Serial console connection will again be blocked while the storage account firewall is enabled.

The transition should take about 2 weeks to complete for private preview customers. If your organization is currently using the private preview and would like a mitigation during the transition period, please send an email using the same email address you used to request access to the preview.

Thank you for your continued interest in Serial console.

Hey All,

Thank you all for your patience. The transition is now complete and there are some changes to the private preview experience that we have included in the updated documentation.

If you are experiencing issues with the private preview or would like access to the private preview, please send an email to azserialhelp@microsoft.com and someone from the Serial console team will get back to you.

[kof-f]

UPDATE: We are happy to announce that we are expecting to have Serial console compatibility with custom storage account firewalls publicly available by the end of June 2022!

[zbyszkogasiorowski]

UPDATE: We are happy to announce that we are expecting to have Serial console compatibility with custom storage account firewalls publicly available by the end of June 2022!

Great news! It will be fantastic if you give us a note when it's ready. Thank you! Kind regards Zbyszko Gasiorowski

[jquinn99002]

Do we have an update on when this will be available? It was expected end of June 2022 but now were near the end of July, can you provide a revised date of when this should be expected?

[nachosmooth]

Hi all, Do we have the Serial Console and the storage firewall working correctly? It is crazy to lock storage accounts with the Firewall and don't allow access to Azure Services like Azure Serial Console from any way. We have BIG security issues with this bad configuration. Please, consider to share with us the IPs or networks to allow access to the Serial Console or create the rule with your Azure Services to allow them always.

[montyplattner]

@kof-f - Is there an updated ETA for when the feature will be publicly available?

[pinyuko]

I noticed the Azure doc has been updated with how to "Use Serial Console with custom boot diagnostics storage account firewall enabled" in https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-linux#serial-console-security

[cata008]

Are there any plans to enable Boot diagnostics to work with a storage account that uses a private endpoint?

[kraduk]

This still doesnt work for us with or without a private ip. The vm containers get created and written to fine, if you add your regions serial public ips to the acl. The issue is with the client gui in the browser. The access always comes from a 10.224/16 address which fails the acl check, and as you cant add rfc1918 to the acl.... I could get 8.0.0.0/6 into the acl which should cover the 10/0 range, however it still didnt work.

[saartanel]

Any news on this issue?

I've tried adding the service endpoints brought out under https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-linux#use-serial-console-with-custom-boot-diagnostics-storage-account-firewall-enabled to my Storage Account's firewall PIP allow list, but it still throws me "forbidden" on my Serial Console connection.

I've also tried adding my VM's subnets under the Storage Accounts allowed Virtual Network, but to no avail.

If I allow all networks on my Storage Account, then it works, but that is obviously an security risk we are not willing to take.

I also enabled diagnostics on my Storage Account to see if I can pinpoint the address that is trying to access for Serial Console connectivity and I can see this kind of request from the diagnostic Log Analytics Workspace "StorageBlobLogs" table:

Location northeurope
Protocol HTTPS
OperationName GetBlob
AuthenticationType SAS
StatusCode 404
StatusText BlobNotFound
Uri https://<redacted>.blob.core.windows.net/bootdiagnostics-redacted0-0444c4b5-4e6b-4d53-98e6-8defd5112374/<redacted>.0444c4b5-4e6b-4d53-98e6-8defd5112374.serialconsole-connectionmetadata?sv=2018-03-28&sr=c&sk=system-1&se=9999-01-01T00%3a00%3a00Z&sp=rwd&sig=XXXXX
CallerIpAddress 10.87.196.12
SchemaVersion 1.0
UserAgentHeader AzSerialConsoleSvcPF
ServiceType blob
Category StorageRead
TlsVersion TLS 1.2
MetricResponseType ClientOtherError
AccessTier None
SourceSystem Azure
Type StorageBlobLogs

To note, this "10.87.196.12" address is not used by the Virtual Machine or not a part of the VM's VNET subnet.

[kraduk]

I saw th same. It's a azure internal infrastructure IP. If they allowed rfc1918 in the ACL all would be good, but alias... I did try with 8.0.0.0/6, and that passed the API tests. The IP was still denied though,

On Fri, 16 Feb 2024, 12:24 Tanel Saar, @.***> wrote:

Any news on this issue?

I've tried adding the service endpoints brought out under https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-linux#use-serial-console-with-custom-boot-diagnostics-storage-account-firewall-enabled to my Storage Account's firewall PIP allow list, but it still throws me "forbidden" on my Serial Console connection.

If I allow all networks on my Storage Account, then it works, but that is obviously an security risk we are not willing to take.

I also enabled diagnostics on my Storage Account to see if I can pinpoint the address that is trying to access for Serial Console connectivity and I can see this kind of request from the diagnostic Log Analytics Workspace "StorageBlobLogs" table:

Location northeurope Protocol HTTPS OperationName GetBlob AuthenticationType SAS StatusCode 404 StatusText BlobNotFound Uri https://.blob.core.windows.net/bootdiagnostics-redacted0-0444c4b5-4e6b-4d53-98e6-8defd5112374/.0444c4b5-4e6b-4d53-98e6-8defd5112374.serialconsole-connectionmetadata?sv=2018-03-28&sr=c&sk=system-1&se=9999-01-01T00%3a00%3a00Z&sp=rwd&sig=XXXXX CallerIpAddress 10.87.196.12 SchemaVersion 1.0 UserAgentHeader AzSerialConsoleSvcPF ServiceType blob Category StorageRead TlsVersion TLS 1.2 MetricResponseType ClientOtherError AccessTier None SourceSystem Azure Type StorageBlobLogs

To note, this "10.87.196.12" address is not used by the Virtual Machine or not a part of the VM's VNET subnet.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/azserialconsole/issues/48#issuecomment-1948293790, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADE23XKKW4O3ULWODA4NNEDYT5FYHAVCNFSM4E4SMEFKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJUHAZDSMZXHEYA . You are receiving this because you commented.Message ID: @.***>

[ondrejholas]

I've tried adding the service endpoints brought out under https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-linux#use-serial-console-with-custom-boot-diagnostics-storage-account-firewall-enabled to my Storage Account's firewall PIP allow list, but it still throws me "forbidden" on my Serial Console connection.

I've also tried adding my VM's subnets under the Storage Accounts allowed Virtual Network, but to no avail.

If I allow all networks on my Storage Account, then it works, but that is obviously an security risk we are not willing to take.

Same here. It used to work before half of February 2024, then it suddenly started to behave as you described - in SA log there are accesses from private IP address, which is not contained in any of our VNETs, and with SA firewall set up in restrictive mode the serial console does not work, even if appropriate public IP addresses are allowed. We are also in North Europe datacenter.

[Tapan1684]

the serial console on Private endpoint stopped working.. seems below change is not working. does anyone have any clue on this . In the current serial console operation, the web socket is opened to an endpoint like

.gateway.serialconsole.azure.com . Ensure the endpoint serialconsole.azure.com is allowed for browser clients in your organization.

[georgejdli]

I noticed that Microsoft added a new IP address for the US regions: 20.83.222.100 https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-linux#use-serial-console-with-custom-boot-diagnostics-storage-account-firewall-enabled

once I added this new IP address to my Storage Account Firewall I was able to access the serial console for my VM again

[kraduk]

that doesn't help when you are seeing rfc1918 addresses

On Tue, 26 Mar 2024 at 16:05, George Li @.***> wrote:

I noticed that Microsoft added a new IP address for the US regions: 20.83.222.100

https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-linux#use-serial-console-with-custom-boot-diagnostics-storage-account-firewall-enabled

once I added this new IP address to my Storage Account Firewall I was able to access the serial console for my VM again

— Reply to this email directly, view it on GitHub https://github.com/microsoft/azserialconsole/issues/48#issuecomment-2020841480, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADE23XIM6UZ2YJMB2JUBKWDY2GMELAVCNFSM4E4SMEFKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBSGA4DIMJUHAYA . You are receiving this because you commented.Message ID: @.***>

[Mekawy29]

Here is the solution that worked with me: Changing Boot Diagnostic Settings from custom storage account to a managed storage account and it's the recommended from Microsoft. So, if you want to use custom storage account you will still facing this issue.

Note: Serial Console is compatible with a managed boot diagnostics storage account. Back to the below doc to check this note. https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/serial-console-linux#serial-console-security