Azure Native v2 schema missing a lot of resources in the default module

[mikhailshilkov]

I’m running some checks on it and it doesn’t seems quite right. There are still many resources that exist in explicit versions but not in the default version. For example, azure-native:workloads/v20211201preview:WordpressInstance exists but azure-native:workloads:WordpressInstance isn’t there. Most of them seem to come from preview versions, but there are stable too, e.g. azure-native:elastic/v20200701:Monitor and no azure-native:elastic:Monitor.

The full list is here: https://gist.github.com/mikhailshilkov/4ca23c2f1ac474de10c945c8ea5a7c04

[kpitzen]

Agreed - we should investigate why this discrepancy is happening and how to resolve. I've added it to the work items required for the 2.0 epic.

[danielrbradley]

The missing resources were due to the v1 removals being applied to v2. This is now solved.

For resources that only exist in the latest preview version, but we're currently pinned at the previous stable version we've not yet determined if we want to pull these in or wait a little longer in case the stable version comes along.

[mikhailshilkov]

For resources that only exist in the latest preview version, but we're currently pinned at the previous stable version we've not yet determined if we want to pull these in or wait a little longer in case the stable version comes along.

Can you generate a list of those?

[danielrbradley]

This is surfaced in the v2-config based on the expect-tracking - if it's not set to preview then it's expected to be a stable version. Approximately 58 services are expected to resolve to a preview version - all others should be stable.

There's also a few services where we specifically exclude preview version from consideration for the default version (search for preview: exclude:

• Cache
• Cdn
• Sql
• StreamAnalytics
• TimeSeriesInsights
• Web

[mikhailshilkov]

@danielrbradley I don't know how to get from v2-config to the list of missing resources in the default API version. I'm trying to see why they are okay to skip. Services like Sql have a comment "Regular stable versions containing all resources" so they shouldn't affect the missing default version.

Or should I run my comparison on schema files again?

[danielrbradley]

@mikhailshilkov can you re-run your analysis as lots of things have changed since you generated this. Also, can you cross check anything you're having flagged that it's not explicitly excluded with a note in the v2-config. Thanks

[mikhailshilkov]

Here is the latest file (on current 2.0-pre HEAD): https://gist.github.com/mikhailshilkov/2a8e9a0939e803d3f05d34a2dbb6eb55

It looks like the list got longer, not shorter...

I spot-checked a few items, e.g. azure-native:synapse:ReadWriteDatabase exists in 1.0 but not in 2.0-pre. Sounds like a no-go?

I'm not fluent with v2-config.yaml but it has no config for Synapse...

[danielrbradley]

No config just means it'll track the latest version and add any resources that only exist in previous versions.

For synapse:ReadWriteDatabase I'm not seeing any resource in the spec called ReadWriteDatabase - only a type.

[mikhailshilkov]

Another relevant view - I compared which v1 resources are missing in v2:

#### Resources
- `🔴` "azure-native:aadiam:azureADMetric" missing
- `🔴` "azure-native:alertsmanagement:ActionRuleByName" missing
- `🔴` "azure-native:apimanagement:ApiDiagnosticLogger" missing
- `🔴` "azure-native:apimanagement:DiagnosticLogger" missing
- `🔴` "azure-native:apimanagement:Property" missing
- `🔴` "azure-native:apimanagement:Schema" missing
- `🔴` "azure-native:authorization:AccessReviewHistoryDefinitionById" missing
- `🔴` "azure-native:authorization:AccessReviewScheduleDefinitionById" missing
- `🔴` "azure-native:authorization:PolicyExemption" missing
- `🔴` "azure-native:authorization:ScopeAccessReviewHistoryDefinitionById" missing
- `🔴` "azure-native:authorization:ScopeAccessReviewScheduleDefinitionById" missing
- `🔴` "azure-native:automanage:Account" missing
- `🔴` "azure-native:automanage:ConfigurationProfilePreference" missing
- `🔴` "azure-native:automation:PrivateEndpointConnection" missing
- `🔴` "azure-native:azurestack:LinkedSubscription" missing
- `🔴` "azure-native:batch:Certificate" missing
- `🔴` "azure-native:billing:BillingRoleAssignmentByBillingAccount" missing
- `🔴` "azure-native:billing:BillingRoleAssignmentByDepartment" missing
- `🔴` "azure-native:billing:BillingRoleAssignmentByEnrollmentAccount" missing
- `🔴` "azure-native:botservice:EnterpriseChannel" missing
- `🔴` "azure-native:containerregistry:AgentPool" missing
- `🔴` "azure-native:containerregistry:ConnectedRegistry" missing
- `🔴` "azure-native:containerregistry:ExportPipeline" missing
- `🔴` "azure-native:containerregistry:ImportPipeline" missing
- `🔴` "azure-native:containerregistry:PipelineRun" missing
- `🔴` "azure-native:containerregistry:TaskRun" missing
- `🔴` "azure-native:containerservice:ManagedClusterSnapshot" missing
- `🔴` "azure-native:containerservice:TrustedAccessRoleBinding" missing
- `🔴` "azure-native:costmanagement:CloudConnector" missing
- `🔴` "azure-native:costmanagement:CostAllocationRule" missing
- `🔴` "azure-native:costmanagement:Report" missing
- `🔴` "azure-native:costmanagement:ReportByBillingAccount" missing
- `🔴` "azure-native:costmanagement:ReportByDepartment" missing
- `🔴` "azure-native:costmanagement:ReportByResourceGroupName" missing
- `🔴` "azure-native:databricks:AccessConnector" missing
- `🔴` "azure-native:datamigration:DatabaseMigrationsSqlDb" missing
- `🔴` "azure-native:datamigration:SqlMigrationService" missing
- `🔴` "azure-native:dbformysql:PrivateEndpointConnection" missing
- `🔴` "azure-native:dbformysql:ServerAdministrator" missing
- `🔴` "azure-native:dbformysql:ServerKey" missing
- `🔴` "azure-native:dbformysql:VirtualNetworkRule" missing
- `🔴` "azure-native:dbforpostgresql:ServerAdministrator" missing
- `🔴` "azure-native:dbforpostgresql:ServerKey" missing
- `🔴` "azure-native:dbforpostgresql:ServerSecurityAlertPolicy" missing
- `🔴` "azure-native:dbforpostgresql:VirtualNetworkRule" missing
- `🔴` "azure-native:desktopvirtualization:PrivateEndpointConnectionByHostPool" missing
- `🔴` "azure-native:desktopvirtualization:PrivateEndpointConnectionByWorkspace" missing
- `🔴` "azure-native:documentdb:CassandraResourceCassandraView" missing
- `🔴` "azure-native:documentdb:GraphResourceGraph" missing
- `🔴` "azure-native:eventgrid:EventChannel" missing
- `🔴` "azure-native:eventgrid:PartnerDestination" missing
- `🔴` "azure-native:eventhub:ApplicationGroup" missing
- `🔴` "azure-native:eventhub:NamespaceIpFilterRule" missing
- `🔴` "azure-native:eventhub:NamespaceVirtualNetworkRule" missing
- `🔴` "azure-native:extendedlocation:ResourceSyncRule" missing
- `🔴` "azure-native:hanaonazure:HanaInstance" missing
- `🔴` "azure-native:hybridcompute:PrivateLinkScopedResource" missing
- `🔴` "azure-native:insights:ComponentLinkedStorageAccount" missing
- `🔴` "azure-native:insights:DiagnosticSetting" missing
- `🔴` "azure-native:insights:GuestDiagnosticsSettingsAssociation" missing
- `🔴` "azure-native:insights:ManagementGroupDiagnosticSetting" missing
- `🔴` "azure-native:insights:PrivateEndpointConnection" missing
- `🔴` "azure-native:insights:PrivateLinkScope" missing
- `🔴` "azure-native:insights:PrivateLinkScopedResource" missing
- `🔴` "azure-native:insights:SubscriptionDiagnosticSetting" missing
- `🔴` "azure-native:insights:guestDiagnosticsSetting" missing
- `🔴` "azure-native:iotcentral:PrivateEndpointConnection" missing
- `🔴` "azure-native:kubernetesconfiguration:PrivateEndpointConnection" missing
- `🔴` "azure-native:kubernetesconfiguration:PrivateLinkScope" missing
- `🔴` "azure-native:kusto:EventHubConnection" missing
- `🔴` "azure-native:logic:WorkflowAccessKey" missing
- `🔴` "azure-native:machinelearning:CommitmentPlan" missing
- `🔴` "azure-native:machinelearningservices:ACIService" missing
- `🔴` "azure-native:machinelearningservices:AKSService" missing
- `🔴` "azure-native:machinelearningservices:EndpointVariant" missing
- `🔴` "azure-native:machinelearningservices:EnvironmentSpecificationVersion" missing
- `🔴` "azure-native:machinelearningservices:LabelingJob" missing
- `🔴` "azure-native:machinelearningservices:LinkedService" missing
- `🔴` "azure-native:machinelearningservices:LinkedWorkspace" missing
- `🔴` "azure-native:machinelearningservices:MachineLearningCompute" missing
- `🔴` "azure-native:machinelearningservices:MachineLearningDataset" missing
- `🔴` "azure-native:machinelearningservices:MachineLearningDatastore" missing
- `🔴` "azure-native:maps:PrivateAtlase" missing
- `🔴` "azure-native:marketplace:PrivateStoreOffer" missing
- `🔴` "azure-native:media:MediaGraph" missing
- `🔴` "azure-native:migrate:MigrateProject" missing
- `🔴` "azure-native:migrate:Solution" missing
- `🔴` "azure-native:mixedreality:ObjectAnchorsAccount" missing
- `🔴` "azure-native:network:DefaultUserRule" missing
- `🔴` "azure-native:network:NetworkSecurityPerimeter" missing
- `🔴` "azure-native:network:NspAccessRule" missing
- `🔴` "azure-native:network:NspAssociation" missing
- `🔴` "azure-native:network:NspProfile" missing
- `🔴` "azure-native:network:SecurityUserConfiguration" missing
- `🔴` "azure-native:network:UserRule" missing
- `🔴` "azure-native:network:UserRuleCollection" missing
- `🔴` "azure-native:network:VirtualNetworkLink" missing
- `🔴` "azure-native:operationalinsights:MachineGroup" missing
- `🔴` "azure-native:security:AlertsSuppressionRule" missing
- `🔴` "azure-native:security:Assignment" missing
- `🔴` "azure-native:security:Automation" missing
- `🔴` "azure-native:security:Connector" missing
- `🔴` "azure-native:security:CustomAssessmentAutomation" missing
- `🔴` "azure-native:security:CustomEntityStoreAssignment" missing
- `🔴` "azure-native:security:IngestionSetting" missing
- `🔴` "azure-native:security:SecurityContact" missing
- `🔴` "azure-native:security:Standard" missing
- `🔴` "azure-native:security:WorkspaceSetting" missing
- `🔴` "azure-native:securityinsights:ActivityCustomEntityQuery" missing
- `🔴` "azure-native:securityinsights:Anomalies" missing
- `🔴` "azure-native:securityinsights:BookmarkRelation" missing
- `🔴` "azure-native:securityinsights:EntityAnalytics" missing
- `🔴` "azure-native:securityinsights:EyesOn" missing
- `🔴` "azure-native:securityinsights:SourceControl" missing
- `🔴` "azure-native:securityinsights:Ueba" missing
- `🔴` "azure-native:servicebus:NamespaceIpFilterRule" missing
- `🔴` "azure-native:servicebus:NamespaceVirtualNetworkRule" missing
- `🔴` "azure-native:servicefabric:Cluster" missing
- `🔴` "azure-native:sql:DisasterRecoveryConfiguration" missing
- `🔴` "azure-native:sql:ServerCommunicationLink" missing
- `🔴` "azure-native:synapse:DatabasePrincipalAssignment" missing
- `🔴` "azure-native:synapse:EventGridDataConnection" missing
- `🔴` "azure-native:synapse:EventHubDataConnection" missing
- `🔴` "azure-native:synapse:IotHubDataConnection" missing
- `🔴` "azure-native:synapse:KustoPoolAttachedDatabaseConfiguration" missing
- `🔴` "azure-native:synapse:KustoPoolPrincipalAssignment" missing
- `🔴` "azure-native:synapse:ReadWriteDatabase" missing
- `🔴` "azure-native:synapse:kustoPool" missing
- `🔴` "azure-native:timeseriesinsights:PrivateEndpointConnection" missing
- `🔴` "azure-native:voiceservices:Contact" missing
- `🔴` "azure-native:workloads:PhpWorkload" missing
- `🔴` "azure-native:workloads:WordpressInstance" missing

[danielrbradley]

Looking at the first missing resource: azure-native:aadiam:azureADMetric these are the API versions available:

{
  "2017-04-01": [
    "DiagnosticSetting"
  ],
  "2017-04-01-preview": [
    "DiagnosticSetting"
  ],
  "2020-03-01": [
    "PrivateEndpointConnection",
    "privateLinkForAzureAd"
  ],
  "2020-03-01-preview": [
    "privateLinkForAzureAd"
  ],
  "2020-07-01-preview": [
    "azureADMetric"
  ]
}

1. There's no specific config for this service - it uses default behaviour - prefer stable
2. This generates a spec to track 2020-03-01 with the addition of DiagnosticSetting from 2017-04-01
3. The generated lock is then missing azureADMetric...

{
  "DiagnosticSetting": "2017-04-01",
  "PrivateEndpointConnection": "2020-03-01",
  "privateLinkForAzureAd": "2020-03-01"
}

Our current strategy is to not consider previews which come soon after a stable version. The adjustment we need to make is to include preview versions where there's not been a follow-up stable version within a reasonable window (say 6 months). Our intention is to not immediately jump on a new preview version if there's likely to be a stable version of the preview released within a short time of the preview.

[danielrbradley]

This should now be resolved as of #2503 being merged cc: @mikhailshilkov

[danielrbradley]

Ok, done a little more analysis comparing the actual added resources to the gist...

Not added but listed in gist:

- alertsmanagement:tenantactivitylogalert
- apimanagement:property
- azurestackhci:guestagent
- azurestackhci:hybrididentitymetadatum
- azurestackhci:machineextension
- batch:certificate
- datashare:adlsgen2storageaccountdataset
- datashare:adlsgen2storageaccountdatasetmapping
- datashare:blobstorageaccountdataset
- datashare:blobstorageaccountdatasetmapping
- dbformariadb:serverkey
- dbforpostgresql:servergroup
- dbforpostgresql:serverkey
- healthbot:getbot
- kusto:database
- machinelearningservices:aciservice
- machinelearningservices:aksservice
- machinelearningservices:endpointvariant
- machinelearningservices:machinelearningcompute
- marketplace:privatestoreoffer
- peering:peeringserviceprefix
- securityinsights:awss3dataconnector
- securityinsights:codelessapipollingdataconnector
- securityinsights:codelessuidataconnector
- securityinsights:dynamics365dataconnector
- securityinsights:gcpdataconnector
- securityinsights:iotdataconnector
- securityinsights:ipsyncer
- securityinsights:mlbehavioranalyticsalertrule
- securityinsights:mstidataconnector
- securityinsights:mtpdataconnector
- securityinsights:nrtalertrule
- securityinsights:office365projectdataconnector
- securityinsights:officeatpdataconnector
- securityinsights:officeirmdataconnector
- securityinsights:officepowerbidataconnector
- securityinsights:threatintelligencealertrule
- securityinsights:titaxiidataconnector
- servicefabric:cluster
- subscription:subscriptionalias
- synapse:sqldatabase
- synapse:sqlpoolsv3

Extra additions not mentioned in the gist:

- apicenter:service
- appplatform:apm
- appplatform:containerregistry
- baremetalinfrastructure:azurebaremetalstorageinstance
- containerregistry:buildstep
- containerservice:updaterun
- costmanagement:connector
- costmanagement:report
- costmanagement:reportbybillingaccount
- costmanagement:reportbydepartment
- costmanagement:reportbyresourcegroupname
- datafactory:changedatacapture
- dbformysql:serveradministrator
- dbformysql:virtualnetworkrule
- dbforpostgresql:serveradministrator
- dbforpostgresql:serversecurityalertpolicy
- dbforpostgresql:virtualnetworkrule
- eventgrid:cacertificate
- eventgrid:client
- eventgrid:clientgroup
- eventgrid:namespace
- eventgrid:namespacetopic
- eventgrid:namespacetopiceventsubscription
- eventgrid:permissionbinding
- eventgrid:topicspace
- eventhub:namespaceipfilterrule
- eventhub:namespacevirtualnetworkrule
- hybridcompute:machineruncommand
- insights:guestdiagnosticssetting
- insights:guestdiagnosticssettingsassociation
- iotfirmwaredefense:firmware
- iotfirmwaredefense:workspace
- kusto:eventhubconnection
- logic:workflowaccesskey
- machinelearning:commitmentplan
- migrate:migrateproject
- migrate:solution
- network:dnssecconfig
- operationalinsights:machinegroup
- security:workspacesetting
- servicebus:namespaceipfilterrule
- servicebus:namespacevirtualnetworkrule
- sql:backuplongtermretentionpolicy

[danielrbradley]

The two synapse resources sqldatabase and sqlpoolsv3 are missing because they only exist in 2020-04-01-preview which was deprecated in the early round of v1 deprecations. These were then not included in the next stable version - the v3 suffix was dropped and the sql prefix was dropped.

[danielrbradley]

SecurityInsights missing resources appear to be related to them being resource variants.

• The missing resources are not listed in the "all resources" report.
• There is no special configuration for the spec.
• No versions have been removed.

My hypothesis is that AlertRule and DataConnector are the base resources and since we're upgrading these it causes some of these variants to be removed.

[danielrbradley]

MachineLearningServices has been fairly heavily overhauled and all the docs only point to the new latest version (2023-04-01) so think this is down to renames/internal reworking.

[danielrbradley]

Kusto database is a base variant - the other variants are still there.

[danielrbradley]

Here's a summary of all the remaining missing resources:

alertsmanagement:tenantactivitylogalert - excluded preview due to unmergable types apimanagement:property - renamed in new version azurestackhci:* - excluded due to broken casing in preview. Not documented batch:certificate - deprecated by Azure datashare:* - not listed - possible variant dbformariadb:serverkey - only in 2020-01-01 dbforpostgresql:servergroup - private preview only dbforpostgresql:serverkey - only in 2020-01-01 healthbot:getbot - lowercase typo - excluded kusto:database - removed base variant machinelearningservices:aciservice - not listed - possible variant machinelearningservices:aksservice - not listed - possible variant machinelearningservices:endpointvariant - not listed - possible variant machinelearningservices:machinelearningcompute - renamed in new version marketplace:privatestoreoffer - renamed in new version peering:peeringserviceprefix - renamed in new version securityinsights:* - not listed - probably variants changed during upgrade. servicefabric:cluster - accidentally excluded in config subscription:subscriptionalias - renamed in new version synapse:sqldatabase - renamed in new version synapse:sqlpoolsv3 - renamed in new version

The variants are the only ones I haven't 100% proven. I think we've upgraded the base resource which has then changed which variants are then generated.

[mikhailshilkov]

Okay, sounds good to me, I won't have time to do due diligence better than what you did @danielrbradley. We can close the issue if you think it's solved.