Missing CLI telemetry data

[JaneYan-2016]

Unmatch telemetry data between CLI and ARM

Related command

Az Arcappliance create

Describe the bug

Our team is investigating the Arc appliance deployment scenario using various telemetry data. The deployment involves creating a single node k8s cluster on-prem, install various agents in the k8s, and create an ARM resource which represents this k8s cluster running on-prem.

We noticed that not all the ARM requests to create the resource, we can find matching CLI telemetry where the resource creation is sent. I.e. the expected sequence is:

1. CLI to create ARM resource
2. ARM getting the request to create the resource And we can find data from #2, but missing from #1

To Reproduce

Not seeing CLI telemetry: in [cluster('ddazureclients.kusto.windows.net').database('AzureCli')]

let ['_startTime']=datetime('2022-06-11T16:15:34Z'); let ['_endTime']=datetime('2022-06-14T16:15:34Z'); //let ['_provider']=dynamic(null); RawEventsAzCli | where EventTimestamp >= _startTime | where EventTimestamp <= _endTime | extend cliVersion = tostring(Properties['context.default.azurecli.extensionname']) | where RawCommand has "arcappliance run vmware" or RawCommand has "arcappliance prepare vmware" or RawCommand has "arcappliance deploy vmware" or RawCommand has "arcappliance create vmware" | where AzureSubscriptionId in ("4f74333a-8bc9-4d50-a3f9-d0be5b68c0c5") //| where cliVersion == "arcappliance@0.2.23" //| where ActionResult == "None" | project EventName, EventTimestamp, RawCommand, ActionResult, cliVersion, ResultSummary, EntityType, ExceptionMessage, Params,Properties, SourceJson, StartTime, EndTime, IsMicrosoftInternal, AzureSubscriptionId, MacAddressHash, MachineId, CorrelationId, UserId, ClientRequestId, SessionId, EventId

But can see ARM logs: in [cluster('armprod.kusto.windows.net').database('ARMProd')] HttpIncomingRequests //| where PreciseTimeStamp > ago(25d) and PreciseTimeStamp < ago(3d) | where PreciseTimeStamp > datetime(2022-06-12) and PreciseTimeStamp < datetime(2022-06-14) | where subscriptionId == "4f74333a-8bc9-4d50-a3f9-d0be5b68c0c5" | where operationName contains "Microsoft.ResourceConnector" and httpMethod != "GET"

Expected behavior

CLI telemetry and ARM resource creation should match, since CLI triggers the ARM resource creation.

Environment summary

Additional context

[yonzhan]

@calvinhzy @evelyn-ys please help with the check.

[evelyn-ys]

CLI telemetry will store records in local cache files(cache and cache.1 under $homeFolder/.azure/telemetry/) and send them together when:

• It has been more than 10 mins since last sent
• The cache file has grown up to more than 128K then it will rotate as a result of cache.1.

In this case, we will send all the records in cache.1. So there does exist delay in CLI sending telemetry to application insights. Not sure if there's any other delay in application insights sending events to telemetry.

[JaneYan-2016]

@evelyn-ys , thanks for the information. Does this mean the delay is up to 10 mins? I believe we are seeing data mismatch after one day. Could there be other issues we are seeing?

[evelyn-ys]

Not only time restriction. Telemetry records won't be sent until customer executes CLI commands more than 43/44 times (Usually 43/44 records can make the cache file larger than 128k which will trigger telemetry upload)

In other words, there's no fixed time schedule that the telemetry would surly be sent.