search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.21k stars 3.82k forks source link

Using parse operator with Search-AzGraph from Az.ResourceGraph module results produces error #13762

Closed cdhgee closed 3 years ago

cdhgee commented 3 years ago

Description

When using the Search-AzGraph cmdlet, it is not possible to use the parse operator. Doing So results in an "invalid query" error:

Search-AzGraph: {
  "error": {
    "code": "BadRequest",
    "message": "Please provide below info when asking for support: timestamp = 2020-12-16T16:57:12.4249513Z, correlationId = 6902d332-f9f0-4f8a-b90c-d29b938836fd.",
    "details": [
      {
        "code": "InvalidQuery",
        "message": "Query is invalid. Please refer to the documentation for the Azure Resource Graph service and fix the error before retrying."
      },
      {
        "code": "InvalidQueryLanguage",
        "message": "InvalidQueryLanguage",
        "line": -1,
        "characterPositionInLine": -1,
        "token": "parse"
      }
    ]
  }
}

Steps to reproduce

Search-AzGraph -Query "Resources | parse id with * '/providers/' resourceProvider '/' resourceType '/' resourceName"

Environment data

Name                           Value
----                           -----
PSVersion                      7.1.0
PSEdition                      Core
GitCommitId                    7.1.0
OS                             Microsoft Windows 10.0.19041
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     0.8.0                 Az.ResourceGraph                    {Search-AzGraph, Get-AzResourceGraphQuery, New-AzResourceGraphQuery, Remove-AzResourceGraphQuery…}

Debug output

❯ Search-AzGraph -Query "Resources | parse id with * '/providers/' resourceProvider '/' resourceType '/' resourceName"
DEBUG: 10:51:17 AM - SearchAzureRmGraph begin processing with ParameterSet '__AllParameterSets'.
DEBUG: 10:51:17 AM - using account id '--REDACTED--'...
DEBUG: [Common.Authentication]: Authenticating using Account: '--REDACTED--', environment: 'AzureCloud', tenant: '--REDACTED--'
DEBUG: SharedTokenCacheCredential.GetToken invoked. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId:
DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId:  ExpiresOn: 2020-12-16T17:36:56.0000000+00:00
DEBUG: SharedTokenCacheCredential.GetToken invoked. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId:
DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId:  ExpiresOn: 2020-12-16T17:36:56.0000000+00:00
DEBUG: [Common.Authentication]: Received token with LoginType 'User', Tenant: '--REDACTED--', UserId: '--REDACTED--'
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
POST

Absolute Uri:
https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2019-04-01

Headers:
x-ms-client-request-id        : f09eedae-89ce-4a44-b436-31f4cce3deb4
Accept-Language               : en-US

Body:
{
  "subscriptions": [
    "--REDACTED--"
  ],
  "query": "Resources | parse id with * '/providers/' resourceProvider '/' resourceType '/' resourceName",
  "options": {
    "$top": 100,
    "$skip": 0,
    "resultFormat": "objectArray"
  }
}

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
BadRequest

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-ratelimit-remaining-tenant-resource-requests: 14
x-ms-user-quota-remaining     : 14
x-ms-user-quota-resets-after  : 00:00:05
Server                        : Kestrel
x-ms-ratelimit-remaining-tenant-reads: 11998
x-ms-request-id               : 82464896-b3d0-4d39-9fdb-11fa2a7659a1
x-ms-correlation-request-id   : 82464896-b3d0-4d39-9fdb-11fa2a7659a1
x-ms-routing-request-id       : NORTHCENTRALUS:20201216T165117Z:82464896-b3d0-4d39-9fdb-11fa2a7659a1
X-Content-Type-Options        : nosniff
Date                          : Wed, 16 Dec 2020 16:51:17 GMT

Body:
{
  "error": {
    "code": "BadRequest",
    "message": "Please provide below info when asking for support: timestamp = 2020-12-16T16:51:17.8572473Z, correlationId = 82464896-b3d0-4d39-9fdb-11fa2a7659a1.",
    "details": [
      {
        "code": "InvalidQuery",
        "message": "Query is invalid. Please refer to the documentation for the Azure Resource Graph service and fix the error before retrying."
      },
      {
        "code": "InvalidQueryLanguage",
        "message": "InvalidQueryLanguage",
        "line": -1,
        "characterPositionInLine": -1,
        "token": "parse"
      }
    ]
  }
}

Search-AzGraph: {
  "error": {
    "code": "BadRequest",
    "message": "Please provide below info when asking for support: timestamp = 2020-12-16T16:51:17.8572473Z, correlationId = 82464896-b3d0-4d39-9fdb-11fa2a7659a1.",
    "details": [
      {
        "code": "InvalidQuery",
        "message": "Query is invalid. Please refer to the documentation for the Azure Resource Graph service and fix the error before retrying."
      },
      {
        "code": "InvalidQueryLanguage",
        "message": "InvalidQueryLanguage",
        "line": -1,
        "characterPositionInLine": -1,
        "token": "parse"
      }
    ]
  }
}
DEBUG: AzureQoSEvent: CommandName - Search-AzGraph; IsSuccess - False; Duration - 00:00:00.3095224;; Exception - Microsoft.Azure.Management.ResourceGraph.Models.ErrorResponseException: Operation returned an invalid status code 'BadRequest'
   at Microsoft.Azure.Management.ResourceGraph.ResourceGraphClient.ResourcesWithHttpMessagesAsync(QueryRequest query, Dictionary`2 customHeaders, CancellationToken cancellationToken);
DEBUG: Finish sending metric.
DEBUG: 10:51:18 AM - SearchAzureRmGraph end processing.

Error output

WARNING: Upcoming breaking changes in the cmdlet 'Resolve-AzError' :
The `Resolve-Error` alias will be removed in a future release.  Please change any scripts that use this alias to use `Resolve-AzError` instead.
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.

   HistoryId: 1

Message        : Operation returned an invalid status code 'BadRequest'
StackTrace     :    at Microsoft.Azure.Management.ResourceGraph.ResourceGraphClient.ResourcesWithHttpMessagesAsync(QueryRequest query, Dictionary`2 customHeaders, CancellationToken cancellationToken)
Exception      : Microsoft.Azure.Management.ResourceGraph.Models.ErrorResponseException
InvocationInfo : {Search-AzGraph}
Line           : Search-AzGraph -Query "Resources | parse id with * '/providers/' resourceProvider '/' resourceType '/' resourceName"
Position       : At line:1 char:1
                 + Search-AzGraph -Query "Resources | parse id with * '/providers/' reso …
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 1

The Azure PowerShell team is listening, please let us know how we are doing: https://aka.ms/azpssurvey?Q_CHL=ERROR.
dingmeng-xue commented 3 years ago

The reference doc is Kusto doc. Search-AzGraph doesn't support it. Please reference ResourceGraph doc https://docs.microsoft.com/en-us/azure/governance/resource-graph/first-query-powershell

cdhgee commented 3 years ago

Is there a plan to expand the list of supported KQL operators in the future?

dingmeng-xue commented 3 years ago

Here is supported operators https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/query-language#supported-tabulartop-level-operators.

I involve Resource Graph team to look into this feature request about parse operator

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @chiragg4u.

Issue Details
## Description When using the `Search-AzGraph` cmdlet, it is not possible to use the [parse operator](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/parseoperator). Doing So results in an "invalid query" error: ``` Search-AzGraph: { "error": { "code": "BadRequest", "message": "Please provide below info when asking for support: timestamp = 2020-12-16T16:57:12.4249513Z, correlationId = 6902d332-f9f0-4f8a-b90c-d29b938836fd.", "details": [ { "code": "InvalidQuery", "message": "Query is invalid. Please refer to the documentation for the Azure Resource Graph service and fix the error before retrying." }, { "code": "InvalidQueryLanguage", "message": "InvalidQueryLanguage", "line": -1, "characterPositionInLine": -1, "token": "parse" } ] } } ``` ## Steps to reproduce ``` Search-AzGraph -Query "Resources | parse id with * '/providers/' resourceProvider '/' resourceType '/' resourceName" ``` ## Environment data ``` Name Value ---- ----- PSVersion 7.1.0 PSEdition Core GitCommitId 7.1.0 OS Microsoft Windows 10.0.19041 Platform Win32NT PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…} PSRemotingProtocolVersion 2.3 SerializationVersion 1.1.0.1 WSManStackVersion 3.0 ``` ## Module versions ``` ModuleType Version PreRelease Name ExportedCommands ---------- ------- ---------- ---- ---------------- Script 0.8.0 Az.ResourceGraph {Search-AzGraph, Get-AzResourceGraphQuery, New-AzResourceGraphQuery, Remove-AzResourceGraphQuery…} ``` ## Debug output ``` ❯ Search-AzGraph -Query "Resources | parse id with * '/providers/' resourceProvider '/' resourceType '/' resourceName" DEBUG: 10:51:17 AM - SearchAzureRmGraph begin processing with ParameterSet '__AllParameterSets'. DEBUG: 10:51:17 AM - using account id '--REDACTED--'... DEBUG: [Common.Authentication]: Authenticating using Account: '--REDACTED--', environment: 'AzureCloud', tenant: '--REDACTED--' DEBUG: SharedTokenCacheCredential.GetToken invoked. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId: DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId: ExpiresOn: 2020-12-16T17:36:56.0000000+00:00 DEBUG: SharedTokenCacheCredential.GetToken invoked. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId: DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId: ExpiresOn: 2020-12-16T17:36:56.0000000+00:00 DEBUG: [Common.Authentication]: Received token with LoginType 'User', Tenant: '--REDACTED--', UserId: '--REDACTED--' DEBUG: ============================ HTTP REQUEST ============================ HTTP Method: POST Absolute Uri: https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2019-04-01 Headers: x-ms-client-request-id : f09eedae-89ce-4a44-b436-31f4cce3deb4 Accept-Language : en-US Body: { "subscriptions": [ "--REDACTED--" ], "query": "Resources | parse id with * '/providers/' resourceProvider '/' resourceType '/' resourceName", "options": { "$top": 100, "$skip": 0, "resultFormat": "objectArray" } } DEBUG: ============================ HTTP RESPONSE ============================ Status Code: BadRequest Headers: Cache-Control : no-cache Pragma : no-cache Strict-Transport-Security : max-age=31536000; includeSubDomains x-ms-ratelimit-remaining-tenant-resource-requests: 14 x-ms-user-quota-remaining : 14 x-ms-user-quota-resets-after : 00:00:05 Server : Kestrel x-ms-ratelimit-remaining-tenant-reads: 11998 x-ms-request-id : 82464896-b3d0-4d39-9fdb-11fa2a7659a1 x-ms-correlation-request-id : 82464896-b3d0-4d39-9fdb-11fa2a7659a1 x-ms-routing-request-id : NORTHCENTRALUS:20201216T165117Z:82464896-b3d0-4d39-9fdb-11fa2a7659a1 X-Content-Type-Options : nosniff Date : Wed, 16 Dec 2020 16:51:17 GMT Body: { "error": { "code": "BadRequest", "message": "Please provide below info when asking for support: timestamp = 2020-12-16T16:51:17.8572473Z, correlationId = 82464896-b3d0-4d39-9fdb-11fa2a7659a1.", "details": [ { "code": "InvalidQuery", "message": "Query is invalid. Please refer to the documentation for the Azure Resource Graph service and fix the error before retrying." }, { "code": "InvalidQueryLanguage", "message": "InvalidQueryLanguage", "line": -1, "characterPositionInLine": -1, "token": "parse" } ] } } Search-AzGraph: { "error": { "code": "BadRequest", "message": "Please provide below info when asking for support: timestamp = 2020-12-16T16:51:17.8572473Z, correlationId = 82464896-b3d0-4d39-9fdb-11fa2a7659a1.", "details": [ { "code": "InvalidQuery", "message": "Query is invalid. Please refer to the documentation for the Azure Resource Graph service and fix the error before retrying." }, { "code": "InvalidQueryLanguage", "message": "InvalidQueryLanguage", "line": -1, "characterPositionInLine": -1, "token": "parse" } ] } } DEBUG: AzureQoSEvent: CommandName - Search-AzGraph; IsSuccess - False; Duration - 00:00:00.3095224;; Exception - Microsoft.Azure.Management.ResourceGraph.Models.ErrorResponseException: Operation returned an invalid status code 'BadRequest' at Microsoft.Azure.Management.ResourceGraph.ResourceGraphClient.ResourcesWithHttpMessagesAsync(QueryRequest query, Dictionary`2 customHeaders, CancellationToken cancellationToken); DEBUG: Finish sending metric. DEBUG: 10:51:18 AM - SearchAzureRmGraph end processing. ``` ## Error output ``` WARNING: Upcoming breaking changes in the cmdlet 'Resolve-AzError' : The `Resolve-Error` alias will be removed in a future release. Please change any scripts that use this alias to use `Resolve-AzError` instead. Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell. HistoryId: 1 Message : Operation returned an invalid status code 'BadRequest' StackTrace : at Microsoft.Azure.Management.ResourceGraph.ResourceGraphClient.ResourcesWithHttpMessagesAsync(QueryRequest query, Dictionary`2 customHeaders, CancellationToken cancellationToken) Exception : Microsoft.Azure.Management.ResourceGraph.Models.ErrorResponseException InvocationInfo : {Search-AzGraph} Line : Search-AzGraph -Query "Resources | parse id with * '/providers/' resourceProvider '/' resourceType '/' resourceName" Position : At line:1 char:1 + Search-AzGraph -Query "Resources | parse id with * '/providers/' reso … + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ HistoryId : 1 The Azure PowerShell team is listening, please let us know how we are doing: https://aka.ms/azpssurvey?Q_CHL=ERROR. ```
Author: cdhgee
Assignees: -
Labels: `Resource Graph`, `Service Attention`, `customer-reported`, `feature-request`
Milestone: -
chiragg4u commented 3 years ago

Thanks for the feedback we are continuously adding more support. Please provide feedback in our uservoice channel so that we can prioritize most important features @ https://feedback.azure.com/forums/915958-azure-governance?category_id=345061

dingmeng-xue commented 3 years ago

@chiragg4u , please share more specific to this feature ask? Is there any workaround?

chiragg4u commented 3 years ago

Parse operator and other generic KUSTO operators are not supported. ARG is a Resource inventory solution and we are using KUSTO as a base and continuously adding more support. I would like you to provide feedback on our uservoice channel so that we can prioritize most important features @ https://feedback.azure.com/forums/915958-azure-governance?category_id=345061

csahut commented 2 years ago

Ok, so Search-AzGraph implements a subset of ARG KQL (https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/query-language#supported-tabulartop-level-operators) which is already a subset of KQL (https://docs.microsoft.com/en-us/azure/data-explorer/kql-quick-reference)

That's a nightmare. Is there any plan to at least have ARG KQL working with Search-AzGraph? I have now a query using "parse" working in the portal but not in Powershell :(

throckmedia commented 2 years ago

From the documentation of supported tabular operators, I see parse in the list, however, I still get the same error provided by the OP that parse is not supported as part of search-azgraph. Has this been fixed or remediated in any way since this thread has been created. The error code still says "parse" for my query through search-azgraph, but the portal works fine.