search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.22k stars 3.83k forks source link

Get-AzOperationalInsightsTable :: doesn't list the tables within the workspace #23784

Closed francesco1119 closed 8 months ago

francesco1119 commented 9 months ago

Description

I was looking at the Get-AzOperationalInsightsTable official documentation and it states that the command:

"[...] list tables under workspace when "-TableName" was not provided."

let's see if it's true, if you run

Get-AzOperationalInsightsTable -ResourceGroupName MyResourceName -WorkspaceName MyWorkspaceName | Select-Object -ExpandProperty Name | sort

the terminal will just print the complete list of all available tables in Log Analytics. Not the tables that are inside that workspace.

the problem goes even further if you use the the command Update-AzOperationalInsightsTable: in that case PowerShell starts updating tables that don't actually exists inside the workspace.

Issue script & Debug output

Get-AzOperationalInsightsTable -ResourceGroupName MyResourceName  -WorkspaceName MyWorkspaceName | Select-Object -ExpandProperty Name | sort

AACAudit
AACHttpRequest
AADB2CRequestLogs
AADCustomSecurityAttributeAuditLogs
....
....
....
etc, 
etc, 
etc,
....
....
WVDErrors
WVDFeeds
WVDHostRegistrations
WVDManagement
WVDSessionHostManagement

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.22621.2506
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.22621.2506
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Module versions

ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     2.10.1     Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clea...
Script     3.2.0      Az.OperationalInsights              {Disable-AzOperationalInsightsIISLogColl...

Error output

WARNING: Upcoming breaking changes in the cmdlet 'Resolve-AzError' :
The `Resolve-Error` alias will be removed in a future release.  Please change any scripts that use
this alias to use `Resolve-AzError` instead.
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning,
and other information on breaking changes in Azure PowerShell.

The Azure PowerShell team is listening, please let us know how we are doing: https://aka.ms/azpssurvey?Q_CHL=ERROR.
isra-fel commented 9 months ago

@VeryEarly please try to reproduce this issue.

NoriZC commented 9 months ago

Hi @francesco1119 could you share the resource id of the tables which in your opinion is 'available tables in Log Analytics but not the tables that are inside that workspace'? I can also get a great number of tables when running this command, but they are all under the certain workspace, and this is the reason why they can be updated.

I compared with the result in portal, (I guess that your did this as well). It seems portal filtered out some of the results. And we should contact service team to ask about the filtering rules, and decide whether to filter the results by default.

francesco1119 commented 9 months ago

I made a video so you can see it.

Log Analytics

I created a dumb Resource Group (rg-dev-westeurope) and Workspace (log-dev-westeurope).

As you can see from the video in that Workspace there are a bunch of tables, if I scroll down at the very bottom I only have 1 that start with the letter "W" and 3 that starts with the letter "U": 

Update
UpdateSummary
Usage
WindowsFirewall

Then I go online and I run the command:

Get-AzOperationalInsightsTable -ResourceGroupName rg-dev-westeurope -WorkspaceName log-dev-westeurope | Select-Object -ExpandProperty Name | sort

And as you can see this is the list of tables that I receive in the letter "U" and "W": 

UCClient
UCClientReadinessStatus
UCClientUpdateStatus
UCDeviceAlert
UCDOAggregatedStatus
UCDOStatus
UCServiceUpdateStatus
UCUpdateAlert
Update
UpdateSummary
Usage
VCoreMongoRequests
VIAudit
VIIndexing
VMBoundPort
VMComputer
VMConnection
VMProcess
W3CIISLog
WebPubSubConnectivity
WebPubSubHttpRequest
WebPubSubMessaging
Windows365AuditLogs
WindowsClientAssessmentRecommendation
WindowsFirewall
WindowsServerAssessmentRecommendation
WorkloadDiagnosticLogs
WVDAgentHealthStatus
WVDAutoscaleEvaluationPooled
WVDCheckpoints
WVDConnectionGraphicsDataPreview
WVDConnectionNetworkData
WVDConnections
WVDErrors
WVDFeeds
WVDHostRegistrations
WVDManagement
WVDSessionHostManagement

The documentation said that:

"[...] list tables under workspace when "-TableName" was not provided."

I'm not providing "-TableName" so it should list all the tables within that Workspace. But instead it's showing me all the passible tables, even those that don't exists.

NoriZC commented 9 months ago

@francesco1119 Yes I got your point. But you are using the tables showed in portal as the standard. And what we focus on is the service response.

If you press F12 when listing the tables in portal, you would see more tables than it showed on the web page in respond body.

Hopefully you can help check whether there are tables which don't really belongs to this workspace when using the -workspace parameter. Could you please run the Get-AzOperationalInsightsTable command with a -debug at the end? By doing so we can see the response body.

francesco1119 commented 9 months ago

But I'm using the -WorkspaceName log-dev-westeurope parameter:

Get-AzOperationalInsightsTable -ResourceGroupName rg-dev-westeurope -WorkspaceName log-dev-westeurope | Select-Object -ExpandProperty Name | sort

I tried to run:

Get-AzOperationalInsightsTable -ResourceGroupName rg-dev-westeurope -WorkspaceName log-dev-westeurope | Select-Object -ExpandProperty Name | sort -debug

but the result is exactly the same.

NoriZC commented 9 months ago

Could you please try Get-AzOperationalInsightsTable -ResourceGroupName rg-dev-westeurope -WorkspaceName log-dev-westeurope -debug | Select-Object -ExpandProperty Name | sort

francesco1119 commented 9 months ago

It's crazy, it's giving me a looong error: 

DEBUG: 18:10:56 - GetAzureOperationalInsightsTableCommand begin processing with ParameterSet 'ByWorkspaceName'.

Confirm
Continue with this operation?
[Y] Yes  [A] Yes to All  [H] Halt Command  [S] Suspend  [?] Help (default is "Y"): A
DEBUG: 18:10:59 - using account id 'f.mantovani@something.com'...
DEBUG: 18:10:59 - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value
 [True].
DEBUG: [Common.Authentication]: Authenticating using Account: 'f.mantovani@something.com', environment: 'AzureCloud', tenant:
'363c9387-f81b-4c87-8f64-93d71a8c1e9b'
DEBUG: 18:10:59 - [SilentAuthenticator] Calling SharedTokenCacheCredential.GetTokenAsync -
TenantId:'363c9387-f81b-4c87-8f64-93d71a8c1e9b', Scopes:'https://management.core.windows.net//.default',
AuthorityHost:'https://login.microsoftonline.com/', UserId:'f.mantovani@something.com'
DEBUG: SharedTokenCacheCredential.GetToken invoked. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId:
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.67 - 39a219c2-ed97-4f13-b6b5-5be477510080]
IsLegacyAdalCacheEnabled: yes
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - 39a219c2-ed97-4f13-b6b5-5be477510080]
[Region discovery] Not using a regional authority.
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - 39a219c2-ed97-4f13-b6b5-5be477510080]
[Region discovery] Not using a regional authority.
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - 39a219c2-ed97-4f13-b6b5-5be477510080]
IsLegacyAdalCacheEnabled: yes
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - 39a219c2-ed97-4f13-b6b5-5be477510080]
IsLegacyAdalCacheEnabled: yes
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68] Found 1 cache accounts and 0 broker accounts
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68] Returning 1 accounts
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e] MSAL
MSAL.Desktop with assembly version '4.39.0.0'. CorrelationId(ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e)
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e] ===
AcquireTokenSilent Parameters ===
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]
LoginHint provided: False
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]
Account provided: True
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]
ForceRefresh: False
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]
=== Request Data ===
Authority Provided? - True
Scopes - https://management.core.windows.net//.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenSilent
IsConfidentialClient - False
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e
UserAssertion set: False
LongRunningOboCacheKey set: False
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e] ===
Token Acquisition (SilentRequest) started:
  Scopes: https://management.core.windows.net//.default
 Authority Host: login.microsoftonline.com
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]
[Region discovery] Not using a regional authority.
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]
Access token is not expired. Returning the found cache entry. [Current time (01/03/2024 17:10:59) - Expiration Time (01/03/2024
18:29:17 +00:00) - Extended Expiration Time (01/03/2024 18:29:17 +00:00)]
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]
Returning access token found in cache. RefreshOn exists ? False
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]
[Region discovery] Not using a regional authority.
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]
Fetched access token from host login.microsoftonline.com.
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]
 === Token Acquisition finished successfully:
DEBUG: False MSAL 4.39.0.0 MSAL.Desktop 4.8 or later Windows 10 Pro [01/03 17:10:59.68 - ffd9fe9e-e984-4dc4-9799-a4d8dde4b70e]  AT
expiration time: 1/3/2024 18:29:17 +00:00, scopes https://management.core.windows.net//user_impersonation
https://management.core.windows.net//.default source Cache from login.microsoftonline.com appHashCode 28181301
DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId:
ExpiresOn: 2024-01-03T18:29:17.0000000+00:00
DEBUG: [Common.Authentication]: Received token with LoginType 'User', Tenant: '363c9387-f81b-4c87-8f64-93d71a8c1e9b', UserId:
'f.mantovani@something.com'
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/f95326b6-759b-45c5-a123-ae6a9aa82754/resourcegroups/rg-dev-westeurope/providers/Microsoft
.OperationalInsights/workspaces/log-dev-westeurope/tables?api-version=2021-12-01-preview

Headers:
x-ms-client-request-id        : 9c521ad4-b873-438b-950b-5596adee6b6d
accept-language               : en-US

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
NotFound

Headers:
Pragma                        : no-cache
x-ms-failure-cause            : gateway
x-ms-request-id               : 994ad4f7-521f-4718-ae74-7af7d23327c5
x-ms-correlation-request-id   : 994ad4f7-521f-4718-ae74-7af7d23327c5
x-ms-routing-request-id       : SWITZERLANDNORTH:20240103T171100Z:994ad4f7-521f-4718-ae74-7af7d23327c5
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
Cache-Control                 : no-cache
Date                          : Wed, 03 Jan 2024 17:11:00 GMT

Body:
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'rg-dev-westeurope' could not be found."
  }
}

DEBUG: 18:11:00 - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning default value [True].

Confirm
Operation returned an invalid status code 'NotFound'
Not Found
{"error":{"code":"ResourceGroupNotFound","message":"Resource group 'rg-dev-westeurope' could not be found."}}
[Y] Yes  [A] Yes to All  [H] Halt Command  [S] Suspend  [?] Help (default is "Y"): A
Get-AzOperationalInsightsTable : Operation returned an invalid status code 'NotFound'
Not Found
{"error":{"code":"ResourceGroupNotFound","message":"Resource group 'rg-dev-westeurope' could not be found."}}
At line:1 char:1
+ Get-AzOperationalInsightsTable -ResourceGroupName "rg-dev-westeurope" ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Get-AzOperationalInsightsTable], Exception
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.OperationalInsights.Tables.GetAzureOperationalInsightsTableCommand

DEBUG: 18:11:22 - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value
 [True].
DEBUG: AzureQoSEvent: Module: Az.OperationalInsights:3.2.0; CommandName: Get-AzOperationalInsightsTable; PSVersion: 5.1.22621.2506;
IsSuccess: False; Duration: 00:00:25.8649404; Exception: Operation returned an invalid status code 'NotFound'
Not Found
{"error":{"code":"ResourceGroupNotFound","message":"Resource group 'rg-dev-westeurope' could not be found."}};
DEBUG: Finish sending metric.
DEBUG: 18:11:23 - GetAzureOperationalInsightsTableCommand end processing.
NoriZC commented 9 months ago

The debug log here indicated that the resource group rg-dev-westeurope could not be found. Could you please re-create the resource group and reproduce the issue you showed, and paste its debug log here?

francesco1119 commented 8 months ago

Tested.

The number of tables returned is always the same. I found a way to put all the debugging into a TXT.

File in attach

out.txt

NoriZC commented 8 months ago

In your debug log, every table showed up has the workspaces/log-dev-westeurope in its id, which means they are in the specified workspace as expected. image

About the difference between Portal and Az Commands, we should involve service team to give some explanation - if tables are filtered in portal UI, what rules are followed to do the filter?

francesco1119 commented 8 months ago

Can you try the other way around?

Try to create a workspace and include only 1 table. Then from the PowerShell list all tables. You will see how all tables will show up.

So what is the point of this command, then?

NoriZC commented 8 months ago

@francesco1119 We have discussed with related experts and it is confirmed that call to list all tables at workspace scope will generally return between 500-1000 tables. It’s a normal behavior. Most of them were created implicitly as a side affect to other resource creation.

PowerShell can not control portal behavior. We are recommanded to reflect the REST experience.

Thanks for your suggestion to make user experience better, but for this case sadly we have nothing to do with. Please include parameter -Name in script to get specific table rather than listing all of them.