New-AzDataCollectionRule. does not create DCR endoints (logs& Metrics)

ChristopheLux commented 1 month ago

Description

Hello Contrary to ARM deployment the DCR logs and metrics endpoints are not generated when created a new DRC with PowerShell

Issue script & Debug output

New-AzDataCollectionRule -ResourceGroupName 'RGxxxxx' -Name 'DCR-ReproTest' -JsonFilePath '/home/azadm/New_DCR_AZPolicyComplianceDetails.json'
DEBUG: 11:45:31 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - GetAzureRMContextCommand begin processing with ParameterSet 'GetSingleContext'.
DEBUG: 11:45:31 AM - [ConfigManager] Got [False] from [DisplayBreakingChangeWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [CheckForUpgrade], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 11:45:31 AM - GetAzureRMContextCommand end processing.
DEBUG: [CmdletBeginProcessing]: Starting command
DEBUG: CmdletBeginProcessing: 
DEBUG: CmdletProcessRecordStart: 
DEBUG: CmdletGetPipeline: 
DEBUG: CmdletBeforeAPICall: 
DEBUG: URLCreated: /subscriptions/yyyyyy/resourceGroups/RGxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01
DEBUG: RequestCreated: /subscriptions/yyyyyy/resourceGroups/RGxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01
DEBUG: HeaderParametersAdded: 
DEBUG: BodyContentSet: 
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [DisableInstanceDiscovery], Module = [], Cmdlet = []. Returning default value [False].
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
PUT

Absolute Uri:
https://management.azure.com/subscriptions/yyyy/resourceGroups/RGxxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01

Headers:
x-ms-unique-id                : 2
x-ms-client-request-id        :yyyyyyyy
CommandName                   : New-AzDataCollectionRule
FullCommandName               : New-AzDataCollectionRule_CreateViaJsonFilePath
ParameterSetName              : __AllParameterSets
User-Agent                    : AzurePowershell/v12.1.0,PSVersion/v7.4.3,Az.DataCollectionRule/5.2.1

Body:
{
  "location": "westeurope",
  "properties": {
    "streamDeclarations": {
      "Custom-Historical_AzPolicyComplianceDetails_CL": {
        "columns": [
          {
            "name": "policyAssignmentId",
            "type": "string"
          },
          {
            "name": "policyDefinitionId",
            "type": "string"
          },
          {
            "name": "policyDefinitionReferenceId",
            "type": "string"
          },
          {
            "name": "policyDefinitionGroupNames",
            "type": "string"
          },
          {
            "name": "policyDefinitionAction",
            "type": "string"
          },
          {
            "name": "numberOfNonCompliantResources",
            "type": "int"
          },
          {
            "name": "numberOfCompliantResources",
            "type": "int"
          },
          {
            "name": "details",
            "type": "dynamic"
          }
        ]
      }
    },
    "destinations": {
      "logAnalytics": [
        {
          "workspaceResourceId": "/subscriptions/xxxxxx/resourceGroups/rgxxxxxxx/providers/microsoft.operationalinsights/workspaces/policyworkspace",
          "workspaceId": "xxxx",
          "name": "myworkspace"
        }
      ]
    },
    "dataFlows": [
      {
        "streams": [
          "Custom-Historical_AzPolicyComplianceDetails_CL"
        ],
        "destinations": [
          "myworkspace"
        ],
        "transformKql": "source\n| extend TimeGenerated = now()\n",
        "outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
      }
    ]
  }
}

DEBUG: BeforeCall: 
DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
Vary                          : Accept-Encoding
x-ms-ratelimit-remaining-subscription-resource-requests: 149
Request-Context               : appId=cid-v1:x
x-ms-correlation-request-id   : x
x-ms-client-request-id        : x
x-ms-routing-request-id       : WESTEUROPE:xx
x-ms-request-id               : xxxx
api-supported-versions        : 2019-11-01-preview, 2021-04-01, 2021-09-01-preview, 2022-06-01, 2023-03-11, 2024-03-11
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
X-Cache                       : CONFIG_NOCACHE
X-MSEdge-Ref                  : Ref A: xxxx Ref B: xxxx Ref C: 2024-08-01T11:45:31Z
Date                          : Thu, 01 Aug 2024 11:45:33 GMT

Body:
{
  "properties": {
    "immutableId": "dcr-ewfwff3",
    "streamDeclarations": {
      "Custom-Historical_AzPolicyComplianceDetails_CL": {
        "columns": [
          {
            "name": "policyAssignmentId",
            "type": "string"
          },
          {
            "name": "policyDefinitionId",
            "type": "string"
          },
          {
            "name": "policyDefinitionReferenceId",
            "type": "string"
          },
          {
            "name": "policyDefinitionGroupNames",
            "type": "string"
          },
          {
            "name": "policyDefinitionAction",
            "type": "string"
          },
          {
            "name": "numberOfNonCompliantResources",
            "type": "int"
          },
          {
            "name": "numberOfCompliantResources",
            "type": "int"
          },
          {
            "name": "details",
            "type": "dynamic"
          }
        ]
      }
    },
    "destinations": {
      "logAnalytics": [
        {
          "workspaceResourceId": "/subscriptions/9yyyyy/resourceGroups/rg-int-dgs-lab-its-itinfra-1/providers/microsoft.operationalinsights/workspaces/policyworkspace",
          "workspaceId": "zzzzzz",
          "name": "myworkspace"
        }
      ]
    },
    "dataFlows": [
      {
        "streams": [
          "Custom-Historical_AzPolicyComplianceDetails_CL"
        ],
        "destinations": [
          "myworkspace"
        ],
        "transformKql": "source\n| extend TimeGenerated = now()\n",
        "outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
      }
    ],
    "provisioningState": "Succeeded"
  },
  "location": "westeurope",
  "id": "/subscriptions/yyyyy/resourceGroups/RGxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest",
  "name": "DCR-ReproTest",
  "type": "Microsoft.Insights/dataCollectionRules",
  "etag": "\"b6009126-0000-0d00-0000-66ab755d0000\"",
  "systemData": {
    "createdBy": "xxxxx",
    "createdByType": "User",
    "createdAt": "2024-08-01T11:45:31.8835297Z",
    "lastModifiedBy": "xxxxx",
    "lastModifiedByType": "User",
    "lastModifiedAt": "2024-08-01T11:45:31.8835297Z"
  }
}

DEBUG: ResponseCreated: 
DEBUG: BeforeResponseDispatch: 
DEBUG: Finally: 
DEBUG: CmdletAfterAPICall: 
DEBUG: [CmdletProcessRecordAsyncEnd]: Finish HTTP process
DEBUG: CmdletProcessRecordAsyncEnd: 
DEBUG: CmdletProcessRecordEnd: 
DEBUG: 11:45:33 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].

DataCollectionEndpointId                  : 
DataFlow                                  : {{
                                              "streams": [ "Custom-Historical_AzPolicyComplianceDetails_CL" ],
                                              "destinations": [ "myworkspace" ],
                                              "transformKql": "source\n| extend TimeGenerated = now()\n",
                                              "outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
                                            }}
DataSourceDataImportEventHubConsumerGroup : 
DataSourceDataImportEventHubName          : 
DataSourceDataImportEventHubStream        : 
DataSourceExtension                       : 
DataSourceIisLog                          : 
DataSourceLogFile                         : 
DataSourcePerformanceCounter              : 
DataSourcePlatformTelemetry               : 
DataSourcePrometheusForwarder             : 
DataSourceSyslog                          : 
DataSourceWindowsEventLog                 : 
DataSourceWindowsFirewallLog              : 
Description                               : 
DestinationAzureMonitorMetricName         : 
DestinationEventHub                       : 
DestinationEventHubsDirect                : 
DestinationLogAnalytic                    : {{
                                              "workspaceResourceId": "/subscriptions/xxxxxxx/resourceGroups/rgxxxxxx/providers/microsoft.operationalinsights/workspaces/policyworkspace",
                                              "workspaceId": "4yyyyy",
                                              "name": "myworkspace"
                                            }}
DestinationMonitoringAccount              : 
DestinationStorageAccount                 : 
DestinationStorageBlobsDirect             : 
DestinationStorageTablesDirect            : 
Etag                                      : "b6009126-0000-0d00-0000-66ab755d0000"
Id                                        : /subscriptions/9XXXXX/resourceGroups/RGXXXXXX1/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest
IdentityPrincipalId                       : 
IdentityTenantId                          : 
IdentityType                              : 
IdentityUserAssignedIdentity              : {
                                            }
ImmutableId                               : dcr-2e40a7469fXXXXXX
Kind                                      : 
Location                                  : westeurope
MetadataProvisionedBy                     : 
MetadataProvisionedByResourceId           : 
Name                                      : DCR-ReproTest
ProvisioningState                         : Succeeded
ResourceGroupName                         : RGXXXX
StreamDeclaration                         : {
                                              "Custom-Historical_AzPolicyComplianceDetails_CL": {
                                                "columns": [
                                                  {
                                                    "name": "policyAssignmentId",
                                                    "type": "string"
                                                  },
                                                  {
                                                    "name": "policyDefinitionId",
                                                    "type": "string"
                                                  },
                                                  {
                                                    "name": "policyDefinitionReferenceId",
                                                    "type": "string"
                                                  },
                                                  {
                                                    "name": "policyDefinitionGroupNames",
                                                    "type": "string"
                                                  },
                                                  {
                                                    "name": "policyDefinitionAction",
                                                    "type": "string"
                                                  },
                                                  {
                                                    "name": "numberOfNonCompliantResources",
                                                    "type": "int"
                                                  },
                                                  {
                                                    "name": "numberOfCompliantResources",
                                                    "type": "int"
                                                  },
                                                  {
                                                    "name": "details",
                                                    "type": "dynamic"
                                                  }
                                                ]
                                              }
                                            }
SystemDataCreatedAt                       : 8/1/2024 11:45:31 AM
SystemDataCreatedBy                       : xxxx
SystemDataCreatedByType                   : User
SystemDataLastModifiedAt                  : 8/1/2024 11:45:31 AM
SystemDataLastModifiedBy                  : xxx
SystemDataLastModifiedByType              : User
Tag                                       : {
                                            }
Type                                      : Microsoft.Insights/dataCollectionRules

DEBUG: AzureQoSEvent:  Module: Az.Monitor:5.2.1; CommandName: New-AzDataCollectionRule; PSVersion: 7.4.3; IsSuccess: True; Duration: 00:00:02.5406602; SanitizeDuration: 00:00:00.0186410

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.3
PSEdition                      Core
GitCommitId                    7.4.3
OS                             CBL-Mariner/Linux
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     3.0.2                 Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     8.0.0                 Az.Compute                          {Add-AzImageDataDisk, Add-AzVhd, Add-AzVMAdditionalUnattendContent, Add-AzVMDataDisk…}
Script     7.8.0                 Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Add-AzApplicationGatewayBackendAddressPool, Add-AzApplicationGatewayBackendHttpSetting, Add-AzApplicationGatewayBackendSetting…}
Script     7.1.0                 Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Script     7.0.0                 Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountManagementPolicyAction, Add-AzStorageAccountNetworkRule, Close-AzStorageFileHandle…}
Script     1.1.3                 Az.Tools.Predictor                  {Disable-AzPredictor, Enable-AzPredictor, Open-AzPredictorSurvey, Send-AzPredictorRating}
Script     0.0.0.10              AzureAD.Standard.Preview            {Add-AzureADApplicationOwner, Add-AzureADDeviceRegisteredOwner, Add-AzureADDeviceRegisteredUser, Add-AzureADDirectoryRoleMember…}
Script     0.9.3                 AzurePSDrive

Error output

No response

JustinGrote commented 1 month ago

Ran into this as well today, I'm pretty sure because it is still using the 2022-06-01 as seen in the trace, and needs to be bumped to the 2023-03-11 api version https://learn.microsoft.com/en-us/rest/api/monitor/data-collection-rules/create?view=rest-monitor-2023-03-11&tabs=HTTP

ChristopheLux commented 1 month ago

Is there any way we can force the API version except going again to Invoke-RestMethod...

JustinGrote commented 1 month ago

@ChristopheLux I made a custom one that uses the newer API version and I still didn't see the endpoint getting populated, so I'm not sure what's going on, I was going to open a ticket and report back.

EDIT: https://gist.github.com/JustinGrote/22c4963f7eb5af08399c26cbf60bc3ae

JustinGrote commented 1 month ago

OK, I think I figured it out.

As of the API spec, there is an ingestion endpoint example where you have to specify the kind as "Direct", note there is a typo, there's an extraneous space in this.

https://learn.microsoft.com/en-us/rest/api/monitor/data-collection-rules/create?view=rest-monitor-2023-03-11&tabs=HTTP#create-or-update-data-collection-rule-with-embedded-ingestion-endpoints

Even though the specs for the Kind parameter say only Windows and Linux are supported values.

I updated my script to specify the kind as Direct, and now I got ingestionEndpoints populated.

    "endpoints": {
      "logsIngestion": "https://xxxx-westus3.logs.z1.ingest.monitor.azure.com",
      "metricsIngestion": "https://xxx-westus3.metrics.z1.ingest.monitor.azure.com"
    },

JustinGrote commented 1 month ago

With some more experimentation in regards to Kind, by supplying invalid data to the API, I get back an error that says these are the actual valid values: Direct,Linux,Windows,WorkspaceTransforms,AgentDirectToStore,AgentSettings,PlatformTelemetry

These appear to be undocumented with a quick google search other than Linux and Windows, these do come back via the 2022 API,

~~and a test of the 2022 API with Direct does populate the endpoints it seems (there's a significant delay, it's not immediate, some sort of provisioning delay)~~ EDIT: Later testing shows this is not the case

and the endpoints can ONLY be seen with the 2023 API.

ChristopheLux commented 1 month ago

Very nice job...I wasn't able to work on this today

ChristopheLux commented 1 month ago

@JustinGrote in the documentation for the PowerShell there is https://learn.microsoft.com/en-us/powershell/module/az.monitor/new-azdatacollectionrule?view=azps-12.1.0 the -Kind. Stupid me

JustinGrote commented 1 month ago

@JustinGrote in the documentation for the PowerShell there is https://learn.microsoft.com/en-us/powershell/module/az.monitor/new-azdatacollectionrule?view=azps-12.1.0 the -Kind. Stupid me

yeah but in my initial testing it doesn't seem to populate endpoints unless the API version is 2023 for the PUT, I'm testing that now.

JustinGrote commented 1 month ago

OK based on this testing with my custom cmdlet:

New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2023Direct' -ApiVersion '2023-03-11'
New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2022Direct' -ApiVersion '2022-06-01'
New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2022DirectReplace' -ApiVersion '2022-06-01'
New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2022DirectReplace' -ApiVersion '2023-03-11' #Overwrites previous

#Additional custom attempt of 2023 API but with Kind not specified at all.

Findings

You must use 2023 API, no 2022 attempts caused the endpoints to populate
You can use PUT or PATCH to update a 2022 API to 2023 and as long as the Kind is Direct, it will get the endpoint property
You must query using 2023 to see the endpoints property, doesn't exist in 2022
Leaving the kind property unspecified does NOT populate the endpoints even with 2023, it looks like you have to use Kind: Direct

Pretty annoying the DCR docs don't mention that Kind: Direct is required

So currently getting DCR endpoints populated is not possible until the API rev gets bumped on this command, you have to use my custom workaround script. I'll updated it and relink https://github.com/Azure/azure-powershell/issues/25727#issuecomment-2265860351

JustinGrote commented 1 month ago

@isra-fel the DataCollectionRule.Autorest needs a bump to 2023-03-11 to resolve this issue.

isra-fel commented 1 month ago

Great findings 👍 Will plan and prioritize this

Azure / azure-powershell