yonzhan
commented
5 months ago Thank you for opening this issue, we will look into it.
Closed Joris-van-der-Wel closed 2 months ago
yonzhan
commented
5 months ago Thank you for opening this issue, we will look into it.
AllyW
commented
5 months ago @Joris-van-der-Wel thanks Joris for this difference. Not sure how azure portal deal with this but azure cli generates code based on service api doc, which can be referred here: https://github.com/Azure/azure-rest-api-specs/blob/42869ad83518d47ab2a24c07d888cb29fdeb43fd/specification/monitor/resource-manager/Microsoft.Insights/stable/2022-06-01/dataCollectionRules_API.json#L1135-L1148 .
If that's the common user case, we'll let service team know the gap and whether they need to remove this enum validations.
Joris-van-der-Wel
commented
5 months ago I think that jsonschema definition is indeed wrong. I think the proper schema would be something like:
{
"streams": {
"description": "List of streams for this data flow.",
"type": "array",
"items": {
"pattern": "^(Microsoft-Event|Microsoft-InsightsMetrics|Microsoft-Perf|Microsoft-Syslog|Microsoft-WindowsEvent|Custom-[\\w-]+)$",
"type": "string"
}
}
}Also, it seems to me that the actual API server uses a different schema to validate with? Would it not be possible to copy that schema? For example if I pass a wrong value to the "streams" property, the server will respond with:
{
"error": {
"code": "InvalidPayload",
"message": "Data collection rule is invalid",
"details": [
{
"code": "InvalidStream",
"message": "'Streams' stream 'Foo' must be a custom stream or one of the allowed streams.",
"target": "Properties.DataFlows[0].Streams[0]"
}
]
}
}Similarly, if I use an invalid property name for the stream declaration, I get this error:
{
"error": {
"code": "InvalidPayload",
"message": "Data collection rule is invalid",
"details": [
{
"code": "InvalidStreamDeclaration",
"message": "'' stream 'Foo' must be a custom stream.",
"target": "Properties.StreamDeclarations[0].Key"
},
{
"code": "InvalidStreamDeclaration",
"message": "'' is not a valid custom stream name. It should start with 'Custom-' followed by a letter and then any combination of alphanumeric characters and '-', '_'",
"target": "Properties.StreamDeclarations[0].Key"
}
]
}
}Custom streams are supported by Azure in all of the other ways that I have tried:
[{"TimeGenerated":"2024-03-28T11:00:00Z","hello":"world"}]example-dcr?api-version=2021-09-01-preview&ignoreMissingTables=true and copy the PUT-dataThis will look something like this:
{
"id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionRules/example-dcr",
"location": "westeurope",
"type": "Microsoft.Insights/dataCollectionRules",
"name": "example-dcr",
"properties": {
"dataSources": {},
"destinations": {
"logAnalytics": [{
"workspaceResourceId": "/subscriptions/XXX/resourcegroups/joris/providers/microsoft.operationalinsights/workspaces/XXX",
"workspaceId": "XXX",
"name": "123456789abcdef123456789abcdef12"
}]
},
"dataFlows": [{
"streams": ["Custom-Example_CL"],
"transformKql": "source",
"destinations": ["123456789abcdef123456789abcdef12"],
"outputStream": "Custom-Example_CL"
}],
"streamDeclarations": {
"Custom-Example_CL": {
"columns": [
{
"name": "TimeGenerated",
"type": "datetime"
},
{
"name": "hello",
"type": "string"
}
]
}
},
"dataCollectionEndpointId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionEndpoints/XXX"
}
}Next, try to use: az monitor data-collection rule list. This will demonstrate the same format for "dataFlows":
[
{
"dataCollectionEndpointId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionEndpoints/XXX",
"dataFlows": [{
"destinations": ["123456789abcdef123456789abcdef12"],
"outputStream": "Custom-Example_CL",
"streams": ["Custom-Example_CL"],
"transformKql": "source"
}],
"dataSources": {},
"destinations": {
"logAnalytics": [{
"name": "123456789abcdef123456789abcdef12",
"workspaceId": "XXX",
"workspaceResourceId": "/subscriptions/XXX/resourcegroups/XXX/providers/microsoft.operationalinsights/workspaces/XXX"
}]
},
"etag": "\"XXX\"",
"id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionRules/example-dcr",
"immutableId": "dcr-3456789abcdef123456789abcdef1245",
"location": "westeurope",
"name": "example-dcr",
"provisioningState": "Succeeded",
"resourceGroup": "XXX",
"streamDeclarations": {
"Custom-Example_CL": {
"columns": [
{
"name": "TimeGenerated",
"type": "datetime"
},
{
"name": "hello",
"type": "string"
}
]
}
},
"systemData": {
"createdAt": "2024-03-28T11:00:00Z",
"createdBy": "jvanderwel@XXX",
"createdByType": "User",
"lastModifiedAt": "2024-03-28T11:00:00Z",
"lastModifiedBy": "jvanderwel@XXX",
"lastModifiedByType": "User"
},
"type": "Microsoft.Insights/dataCollectionRules"
}
]I can deploy a bicep template that uses a custom stream without issue:
resource workspace 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
name: 'myworkspace'
location: 'westeurope'
}
resource dce 'Microsoft.Insights/dataCollectionEndpoints@2022-06-01' = {
name: 'mydce'
location: 'westeurope'
}
resource table 'Microsoft.OperationalInsights/workspaces/tables@2022-10-01' = {
parent: workspace
name: 'Example_CL'
properties: {
schema: {
name: 'Example_CL'
columns: [
{name: 'TimeGenerated', type: 'datetime'}
{name: 'hello', type: 'string'}
]
}
}
}
resource dcr 'Microsoft.Insights/dataCollectionRules@2022-06-01' = {
name: 'mydcr'
location: 'westeurope'
properties: {
dataCollectionEndpointId: dce.id
dataFlows: [
{
destinations: ['log-analytics-dest']
outputStream: 'Custom-Example_CL'
streams: ['Custom-Example_CL']
transformKql: 'source'
}
]
destinations: {
logAnalytics: [
{
name: 'log-analytics-dest'
workspaceResourceId: workspace.id
}
]
}
streamDeclarations: {
'Custom-Example_CL': {
columns: [
{name: 'TimeGenerated', type: 'datetime'}
{name: 'hello', type: 'string'}
]
}
}
}
}@Joris-van-der-Wel we will add support for enumeration arguments which skips validation recently. Please follow the linked pr for future development and releases.
Describe the bug
I am attempting to create a data collection rule in azure monitor, that is intended to be used with a custom application that calls the Logs Ingestion API. I have created a logs-analytics workspace containing a custom table. My data collection rule should be configured without any data sources, instead a custom stream for my table is declared.
This all works fine when I configure this via the Azure Portal GUI. However the CLI refuses to create the data-collection rule because the validation for the
--data-flowsoption is too strict. Specifically, the validation for the"streams"property: https://github.com/Azure/azure-cli-extensions/blob/e2dad8c97f70c6b002749a7864f9dda2485498dd/src/monitor-control-service/azext_amcs/aaz/latest/monitor/data_collection/rule/_create.py#L157C9-L159C10If I comment out line 158 (containing
enum=...):nano ~/.azure/cliextensions/monitor-control-service/azext_amcs/aaz/latest/monitor/data_collection/rule/_create.py. Then my rule is created without issue.Related command
Errors
unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'
Issue script & Debug output
Expected behavior
The rule should be created without error
Environment Summary
Additional context
No response