search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.26k stars 3.86k forks source link

Az.Monitor LogIngestion in Custom Tables: Output stream not generated #21831

Open KarlWindhager opened 1 year ago

KarlWindhager commented 1 year ago

Description

I create Custom Tables in my Log Analytics Workspace. I also create a Data Collection Endpoint and Data Collection Rule. When I create my data collection rule, it is successfully created and so are the Data Sources. However the Destination of each Data Source is empty. I tried to go by this tutorial: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-api

Exporting the template in the Azure Portal lacks the transformKql and the outputstream.

Data Sources Tab of Data Collection Rule in Azure Portal: image

Issue script & Debug output

My Custom Table look like this:
`{
    "properties": {
        "schema": {
            "name": "Karl3_CL",
            "columns": [
  {
    "name": "TimeGenerated",
    "type": "datetime"
  },
  {
    "name": "EventType",
    "type": "string"
  },
  {
    "name": "ApplicationName",
    "type": "string"
  },
  {
    "name": "EventId",
    "type": "string"
  },
  {
    "name": "ApplicationId",
    "type": "string"
  },
  {
    "name": "EventData",
    "type": "string"
  }
]
        }
    }
}`

Template (properties section) used for creation:
`{
    "properties": {
        "dataCollectionEndpointId": "redacted",
        "streamDeclarations": {
  "Custom-Karl3_CL": {
    "columns": [
      {
        "name": "TimeGenerated",
        "type": "datetime"
      },
      {
        "name": "EventType",
        "type": "string"
      },
      {
        "name": "ApplicationName",
        "type": "string"
      },
      {
        "name": "EventId",
        "type": "string"
      },
      {
        "name": "ApplicationId",
        "type": "string"
      },
      {
        "name": "EventData",
        "type": "string"
      }
    ]
  }
},
        "dataSources": {},
        "destinations": {
            "logAnalytics": [
                {
                    "workspaceResourceId": "redacted",
                    "name": "b4414f66b1fc48e7b33a2c9ead545134"
                }
            ]
        },
        "dataFlows": [
  {
    "destinations": [
      "b4414f66b1fc48e7b33a2c9ead545134"
    ],
    "streams": [
      "Custom-Karl3_CL"
    ],
    "transformKql": "source",
    "outputStream": "Custom-Karl3_CL"
  }
]
    }
}
`

Template after creation (from Azure Portal):
`{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionRules_resourcename_name": {
            "defaultValue": "redacted",
            "type": "String"
        },
        "workspaces_resourcename_externalid": {
            "defaultValue": "redacted",
            "type": "String"
        }
    },
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "apiVersion": "2022-06-01",
            "name": "[parameters('dataCollectionRules_resourcename_name')]",
            "location": "West Europe",
            "tags": {
                "Owner": "Karl"
            },
            "properties": {
                "dataSources": {},
                "destinations": {
                    "logAnalytics": [
                        {
                            "workspaceResourceId": "[parameters('workspaces_resourcename_externalid')]",
                            "name": "b4414f66b1fc48e7b33a2c9ead545134"
                        }
                    ]
                },
                "dataFlows": [
                    {
                        "streams": [
                            "Custom-Karl3_CL"
                        ],
                        "destinations": [
                            "b4414f66b1fc48e7b33a2c9ead545134"
                        ]
                    }
                ]
            }
        }
    ]
}`

I create the datacollection rule using this function:

$dataCollectionRule = New-AzDataCollectionRule `
    -ResourceGroupName $ResourceGroupName `
    -RuleName $DataCollectionRuleName `
    -Location $Location `
    -RuleFile $dcrRuleFilePath

Environment data

Name                           Value
----                           -----
PSVersion                      7.3.4
PSEdition                      Core
GitCommitId                    7.3.4
OS                             Microsoft Windows 10.0.19044
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}       
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     9.1.1                 Az
Script     2.10.4                Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     2.0.0                 Az.Advisor                          {Disable-AzAdvisorRecommendation, Enable-AzAdvisorRecommendation, Get-AzAdvisorConfiguration, Get-AzAdvisorRecommendation…}
Script     5.0.1                 Az.Aks                              {Disable-AzAksAddOn, Enable-AzAksAddOn, Get-AzAksCluster, Get-AzAksNodePool…}
Script     1.1.4                 Az.AnalysisServices                 {Add-AzAnalysisServicesAccount, Export-AzAnalysisServicesInstanceLog, Get-AzAnalysisServicesServer, New-AzAnalysisServicesFirewallConfig…}
Script     4.0.1                 Az.ApiManagement                    {Add-AzApiManagementApiToGateway, Add-AzApiManagementApiToProduct, Add-AzApiManagementProductToGroup, Add-AzApiManagementRegion…}
Script     1.2.0                 Az.AppConfiguration                 {Clear-AzAppConfigurationDeletedStore, Get-AzAppConfigurationDeletedStore, Get-AzAppConfigurationStore, Get-AzAppConfigurationStoreKey…}
Script     2.1.0                 Az.ApplicationInsights              {Get-AzApplicationInsights, Get-AzApplicationInsightsApiKey, Get-AzApplicationInsightsContinuousExport, Get-AzApplicationInsightsLinkedStorageAccount…}
Script     2.0.0                 Az.Attestation                      {Add-AzAttestationPolicySigner, Get-AzAttestationPolicy, Get-AzAttestationPolicySigners, Remove-AzAttestationPolicySigner…}
Script     1.8.0                 Az.Automation                       {Export-AzAutomationDscConfiguration, Export-AzAutomationDscNodeReportContent, Export-AzAutomationRunbook, Get-AzAutomationAccount…}
Script     3.2.1                 Az.Batch                            {Disable-AzBatchAutoScale, Disable-AzBatchComputeNodeScheduling, Disable-AzBatchJob, Disable-AzBatchJobSchedule…}
Script     2.0.0                 Az.Billing                          {Get-AzBillingAccount, Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzBillingProfile…}
Script     2.1.0                 Az.Cdn                              {Clear-AzCdnEndpointContent, Clear-AzFrontDoorCdnEndpointContent, Disable-AzCdnCustomDomainCustomHttps, Enable-AzCdnCustomDomainCustomHttps…}
Script     1.1.0                 Az.CloudService                     {Get-AzCloudService, Get-AzCloudServiceInstanceView, Get-AzCloudServiceNetworkInterface, Get-AzCloudServiceOSFamily…}
Script     1.12.0                Az.CognitiveServices                {Add-AzCognitiveServicesAccountNetworkRule, Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountCommitmentPlan, Get-AzCognitiveServicesAccountDeployment…}
Script     5.1.1                 Az.Compute                          {Add-AzImageDataDisk, Add-AzVhd, Add-AzVMAdditionalUnattendContent, Add-AzVMDataDisk…}
Script     1.0.0                 Az.ConfidentialLedger               {Get-AzConfidentialLedger, New-AzConfidentialLedger, New-AzConfidentialLedgerAADBasedSecurityPrincipalObject, New-AzConfidentialLedgerCertBasedSecurityPrincipalObject…}
Script     3.1.0                 Az.ContainerInstance                {Add-AzContainerInstanceOutput, Get-AzContainerGroup, Get-AzContainerInstanceCachedImage, Get-AzContainerInstanceCapability…}
Script     3.0.0                 Az.ContainerRegistry                {Connect-AzContainerRegistry, Get-AzContainerRegistry, Get-AzContainerRegistryCredential, Get-AzContainerRegistryManifest…}
Script     1.8.2                 Az.CosmosDB                         {Get-AzCosmosDBAccount, Get-AzCosmosDBAccountKey, Get-AzCosmosDBCassandraKeyspace, Get-AzCosmosDBCassandraKeyspaceThroughput…}
Script     1.1.0                 Az.DataBoxEdge                      {Get-AzDataBoxEdgeBandwidthSchedule, Get-AzDataBoxEdgeDevice, Get-AzDataBoxEdgeJob, Get-AzDataBoxEdgeOrder…}
Script     1.4.0                 Az.Databricks                       {Get-AzDatabricksAccessConnector, Get-AzDatabricksOutboundNetworkDependenciesEndpoint, Get-AzDatabricksVNetPeering, Get-AzDatabricksWorkspace…}
Script     1.16.10               Az.DataFactory                      {Add-AzDataFactoryV2DataFlowDebugSessionPackage, Add-AzDataFactoryV2TriggerSubscription, Get-AzDataFactory, Get-AzDataFactoryActivityWindow…}
Script     1.0.2                 Az.DataLakeAnalytics                {Add-AzDataLakeAnalyticsDataSource, Add-AzDataLakeAnalyticsFirewallRule, Get-AzDataLakeAnalyticsAccount, Get-AzDataLakeAnalyticsCatalogItem…}
Script     1.3.0                 Az.DataLakeStore                    {Add-AzDataLakeStoreFirewallRule, Add-AzDataLakeStoreItemContent, Add-AzDataLakeStoreTrustedIdProvider, Add-AzDataLakeStoreVirtualNetworkRule…}
Script     1.0.0                 Az.DataProtection                   {Backup-AzDataProtectionBackupInstanceAdhoc, Edit-AzDataProtectionPolicyRetentionRuleClientObject, Edit-AzDataProtectionPolicyTagClientObject, Edit-AzDataProtectionPolicyTriggerClientObject…}
Script     1.0.1                 Az.DataShare                        {Get-AzDataShare, Get-AzDataShareAccount, Get-AzDataShareDataSet, Get-AzDataShareDataSetMapping…}
Script     1.1.0                 Az.DeploymentManager                {Get-AzDeploymentManagerArtifactSource, Get-AzDeploymentManagerRollout, Get-AzDeploymentManagerService, Get-AzDeploymentManagerServiceTopology…}
Script     3.1.1                 Az.DesktopVirtualization            {Disconnect-AzWvdUserSession, Expand-AzWvdMsixImage, Get-AzWvdApplication, Get-AzWvdApplicationGroup…}
Script     1.0.2                 Az.DevTestLabs                      {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoStartPolicy, Get-AzDtlVMsPerLabPolicy…}
Script     1.1.2                 Az.Dns                              {Add-AzDnsRecordConfig, Get-AzDnsRecordSet, Get-AzDnsZone, New-AzDnsRecordConfig…}
Script     1.5.0                 Az.EventGrid                        {Enable-AzEventGridPartnerTopic, Get-AzEventGridChannel, Get-AzEventGridDomain, Get-AzEventGridDomainKey…}
Script     3.1.0                 Az.EventHub                         {Add-AzEventHubIPRule, Add-AzEventHubVirtualNetworkRule, Get-AzEventHubNamespace, New-AzEventHubAuthorizationRuleSASToken…}
Script     1.9.0                 Az.FrontDoor                        {Disable-AzFrontDoorCustomDomainHttps, Enable-AzFrontDoorCustomDomainHttps, Get-AzFrontDoor, Get-AzFrontDoorFrontendEndpoint…}
Script     4.0.6                 Az.Functions                        {Get-AzFunctionApp, Get-AzFunctionAppAvailableLocation, Get-AzFunctionAppPlan, Get-AzFunctionAppSetting…}
Script     5.0.1                 Az.HDInsight                        {Add-AzHDInsightClusterIdentity, Add-AzHDInsightComponentVersion, Add-AzHDInsightConfigValue, Add-AzHDInsightMetastore…}
Script     2.0.0                 Az.HealthcareApis                   {Get-AzHealthcareApisService, Get-AzHealthcareApisWorkspace, Get-AzHealthcareDicomService, Get-AzHealthcareFhirDestination…}
Script     2.7.4                 Az.IotHub                           {Add-AzIotHubCertificate, Add-AzIotHubConfiguration, Add-AzIotHubDeployment, Add-AzIotHubDevice…}
Script     4.9.0                 Az.KeyVault                         {Add-AzKeyVaultCertificate, Add-AzKeyVaultCertificateContact, Add-AzKeyVaultKey, Add-AzKeyVaultManagedStorageAccount…}
Script     2.1.0                 Az.Kusto                            {Add-AzKustoClusterLanguageExtension, Add-AzKustoDatabasePrincipal, Get-AzKustoAttachedDatabaseConfiguration, Get-AzKustoCluster…}
Script     1.5.0                 Az.LogicApp                         {Get-AzIntegrationAccount, Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountAssembly, Get-AzIntegrationAccountBatchConfiguration…}
Script     1.1.3                 Az.MachineLearning                  {Add-AzMlWebServiceRegionalProperty, Export-AzMlWebService, Get-AzMlCommitmentAssociation, Get-AzMlCommitmentPlan…}
Script     1.2.1                 Az.Maintenance                      {Get-AzApplyUpdate, Get-AzConfigurationAssignment, Get-AzMaintenanceConfiguration, Get-AzMaintenancePublicConfiguration…}
Script     1.1.0                 Az.ManagedServiceIdentity           {Get-AzFederatedIdentityCredentials, Get-AzSystemAssignedIdentity, Get-AzUserAssignedIdentity, Get-AzUserAssignedIdentityAssociatedResource…}
Script     3.0.0                 Az.ManagedServices                  {Get-AzManagedServicesAssignment, Get-AzManagedServicesDefinition, Get-AzManagedServicesMarketplaceDefinition, New-AzManagedServicesAssignment…}
Script     2.0.0                 Az.MarketplaceOrdering              {Get-AzMarketplaceTerms, Invoke-AzMarketplaceSignTerms, Set-AzMarketplaceTerms, Stop-AzMarketplaceTerms}
Script     1.1.1                 Az.Media                            {Get-AzMediaService, Get-AzMediaServiceKey, Get-AzMediaServiceNameAvailability, New-AzMediaService…}
Script     2.1.0                 Az.Migrate                          {Get-AzMigrateDiscoveredServer, Get-AzMigrateJob, Get-AzMigrateProject, Get-AzMigrateReplicationFabric…}
Script     4.2.1                 Az.Monitor                          {Add-AzLogProfile, Add-AzMetricAlertRule, Add-AzMetricAlertRuleV2, Add-AzWebtestAlertRule…}
Script     1.1.0                 Az.MySql                            {Get-AzMySqlConfiguration, Get-AzMySqlConnectionString, Get-AzMySqlFirewallRule, Get-AzMySqlFlexibleServer…}
Script     5.1.2                 Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Add-AzApplicationGatewayBackendAddressPool, Add-AzApplicationGatewayBackendHttpSetting, Add-AzApplicationGatewayBackendSetting…}
Script     1.1.1                 Az.NotificationHubs                 {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRule, Get-AzNotificationHubListKey, Get-AzNotificationHubPNSCredential…}
Script     3.2.0                 Az.OperationalInsights              {Disable-AzOperationalInsightsIISLogCollection, Disable-AzOperationalInsightsLinuxCustomLogCollection, Disable-AzOperationalInsightsLinuxPerformanceCollection, Disable-AzOperationalInsightsLinuxSyslogCollection…}
Script     1.5.1                 Az.PolicyInsights                   {Get-AzPolicyEvent, Get-AzPolicyMetadata, Get-AzPolicyRemediation, Get-AzPolicyState…}
Script     1.1.0                 Az.PostgreSql                       {Get-AzPostgreSqlConfiguration, Get-AzPostgreSqlConnectionString, Get-AzPostgreSqlFirewallRule, Get-AzPostgreSqlFlexibleServer…}
Script     1.2.0                 Az.PowerBIEmbedded                  {Get-AzPowerBIEmbeddedCapacity, Get-AzPowerBIWorkspace, Get-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollectionAccessKey…}
Script     1.0.3                 Az.PrivateDns                       {Add-AzPrivateDnsRecordConfig, Get-AzPrivateDnsRecordSet, Get-AzPrivateDnsVirtualNetworkLink, Get-AzPrivateDnsZone…}
Script     6.1.1                 Az.RecoveryServices                 {Add-AzRecoveryServicesAsrReplicationProtectedItemDisk, Backup-AzRecoveryServicesBackupItem, Copy-AzRecoveryServicesVault, Disable-AzRecoveryServicesBackupAutoProtection…}
Script     1.6.0                 Az.RedisCache                       {Export-AzRedisCache, Get-AzRedisCache, Get-AzRedisCacheFirewallRule, Get-AzRedisCacheKey…}
Script     1.1.0                 Az.RedisEnterpriseCache             {Export-AzRedisEnterpriseCache, Get-AzRedisEnterpriseCache, Get-AzRedisEnterpriseCacheDatabase, Get-AzRedisEnterpriseCacheKey…}
Script     1.0.3                 Az.Relay                            {Get-AzRelayAuthorizationRule, Get-AzRelayHybridConnection, Get-AzRelayKey, Get-AzRelayNamespace…}
Script     1.1.0                 Az.ResourceMover                    {Add-AzResourceMoverMoveResource, Get-AzResourceMoverMoveCollection, Get-AzResourceMoverMoveResource, Get-AzResourceMoverRequiredForResources…}
Script     6.4.1                 Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Script     1.3.0                 Az.Security                         {Add-AzSecurityAdaptiveNetworkHardening, Add-AzSecuritySqlVulnerabilityAssessmentBaseline, Confirm-AzSecurityAutomation, Disable-AzIotSecurityAnalyticsAggregatedAlert…}
Script     3.0.0                 Az.SecurityInsights                 {Get-AzSentinelAlertRule, Get-AzSentinelAlertRuleAction, Get-AzSentinelAlertRuleTemplate, Get-AzSentinelAutomationRule…}
Script     2.0.0                 Az.ServiceBus                       {Add-AzServiceBusIPRule, Add-AzServiceBusVirtualNetworkRule, Get-AzServiceBusNamespace, Get-AzServiceBusOperation…}
Script     3.1.0                 Az.ServiceFabric                    {Add-AzServiceFabricClientCertificate, Add-AzServiceFabricManagedClusterClientCertificate, Add-AzServiceFabricManagedNodeTypeVMExtension, Add-AzServiceFabricManagedNodeTypeVMSecret…}
Script     1.4.1                 Az.SignalR                          {Get-AzSignalR, Get-AzSignalRKey, Get-AzSignalRUsage, New-AzSignalR…}
Script     4.1.0                 Az.Sql                              {Add-AzSqlDatabaseToFailoverGroup, Add-AzSqlElasticJobStep, Add-AzSqlElasticJobTarget, Add-AzSqlInstanceKeyVaultKey…}
Script     1.1.0                 Az.SqlVirtualMachine                {Get-AzAvailabilityGroupListener, Get-AzSqlVM, Get-AzSqlVMGroup, New-AzAvailabilityGroupListener…}
Script     1.3.2                 Az.StackHCI                         {Add-AzStackHCIVMAttestation, Disable-AzStackHCIAttestation, Disable-AzStackHCIRemoteSupport, Enable-AzStackHCIAttestation…}
Script     5.1.0                 Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountManagementPolicyAction, Add-AzStorageAccountNetworkRule, Close-AzStorageFileHandle…}
Script     1.7.0                 Az.StorageSync                      {Get-AzStorageSyncCloudEndpoint, Get-AzStorageSyncGroup, Get-AzStorageSyncServer, Get-AzStorageSyncServerEndpoint…}
Script     2.0.0                 Az.StreamAnalytics                  {Get-AzStreamAnalyticsCluster, Get-AzStreamAnalyticsClusterStreamingJob, Get-AzStreamAnalyticsDefaultFunctionDefinition, Get-AzStreamAnalyticsFunction…}
Script     1.0.0                 Az.Support                          {Get-AzSupportProblemClassification, Get-AzSupportService, Get-AzSupportTicket, Get-AzSupportTicketCommunication…}
Script     2.0.0                 Az.Synapse                          {Add-AzSynapseDataFlowDebugSessionPackage, Add-AzSynapseTriggerSubscription, Clear-AzSynapseSqlPoolVulnerabilityAssessmentRuleBaseline, Convert-AzSynapseSqlPoolVulnerabilityAssessmentScan…}
Script     1.1.0                 Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, Add-AzTrafficManagerEndpointConfig, Add-AzTrafficManagerExpectedStatusCodeRange…}
Script     2.11.5                Az.Websites                         {Add-AzWebAppAccessRestrictionRule, Add-AzWebAppTrafficRouting, Edit-AzWebAppBackupConfiguration, Enter-AzWebAppContainerPSSession…}

Error output

No response

ghost commented 1 year ago

Thank you for your feedback. This has been routed to the support team for assistance.

SwathiDhanwada-MSFT commented 1 year ago

@KarlWindhager Thanks for your comment. We will review the issue and get back to you shortly.

KarlWindhager commented 1 year ago

Hi @SwathiDhanwada-MSFT are there any news concercing this issue? Could you reproduced it already?

KarlWindhager commented 1 year ago

I use az 9.7.1 now. It seems to work now without any further changes?

microsoft-github-policy-service[bot] commented 1 year ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @SameergMS, @yairgil.