Error "Collection was modified; enumeration operation may not execute." when dispatching async operations via kusto ICslAdminProvider

[cplankl]

Hello there,

I experienced a problem sending admin commands to Kusto with the SDK using ICslAdminProvider. I use the latest Version 12.0.1 of Microsoft.Azure.Kusto.Data package.

The script I executed via ExecuteControlCommandAsync(...) is the following

.execute database script <|
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2024-01-20T14:39:12.9597558Z) .. datetime(2024-02-19T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-12-21T14:39:12.9597558Z) .. datetime(2024-01-20T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-11-21T14:39:12.9597558Z) .. datetime(2023-12-21T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-10-22T14:39:12.9597558Z) .. datetime(2023-11-21T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-09-22T14:39:12.9597558Z) .. datetime(2023-10-22T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-08-23T14:39:12.9597558Z) .. datetime(2023-09-22T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-07-24T14:39:12.9597558Z) .. datetime(2023-08-23T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-06-24T14:39:12.9597558Z) .. datetime(2023-07-24T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-05-25T14:39:12.9597558Z) .. datetime(2023-06-24T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-04-25T14:39:12.9597558Z) .. datetime(2023-05-25T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-03-26T14:39:12.9597558Z) .. datetime(2023-04-25T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-02-24T14:39:12.9597558Z) .. datetime(2023-03-26T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;
    .append async RawContent <| 
        RawContent
        | where Timestamp between (datetime(2023-01-25T14:39:12.9597558Z) .. datetime(2023-02-24T14:39:12.9597558Z))
        | where Content has <condition>
        | project <multiple columns>;

Basically it appends data into a certain table in time slices.

Sometimes when I execute this command I do not get operation Ids for all the append parts. When I open the data explorer in the Azure portal and use .show operations command one of the operations fails with the error reason: "Collection was modified; enumeration operation may not execute."

Do you have any explanations why this error occurs?

I never experienced this when I executed those commands directly in the Azure portal. Only when using the ICslAdminProvider.

[ohadbitt]

Hi Please provide a ClientActivityId of the .show operations call Additionally could be nice to provide timestamp and cluster name

Also can you say how much is sometimes ? (10% of the times ?) - not so important - only if you have this info .

BTW The error should not be related to which query/command you ran - its a failure in the .show operations itself probably

[cplankl]

Thanks for your fast response.

Here are the details for the operations:

datatable (OperationId: guid, LastUpdatedOn: datetime, Operation: string, NodeId: string, StartedOn: datetime, Duration: timespan, State: string, Status: string, RootActivityId: guid, ShouldRetry: bool, Database: string, AdminEpochStartTime: datetime) [
    guid(3a7610ce-4759-4d2d-8ac4-054625c197aa),datetime(2024-02-19T14:39:12.994207Z),"TableAppend","",datetime(2024-02-19T14:39:12.9785842Z),timespan(00:00:00.0156228),"Failed","Collection was modified; enumeration operation may not execute.",guid(77178bb4-447e-4a6c-b26d-856844479cb3),bool(true),"<database name>",datetime(2024-02-19T09:27:37.1155833Z),
    guid(f442151a-31db-4811-998c-8a610afe8c04),datetime(2024-02-19T14:41:16.1506508Z),"TableAppend","KSENGINE000000",datetime(2024-02-19T14:39:12.9785842Z),timespan(00:02:03.1720666),"Completed","",guid(2a624d29-7a45-4fc4-b33f-672d3842c7d1),bool(false),"<database name>",datetime(2024-02-19T09:27:37.1155833Z)
]

For this kind of scripts (I ran it probably 10-15 times) the error occurred approximately 30% of all times.

[cplankl]

@ohadbitt any information regarding this error?

[ohadbitt]

@cplankl We are still investigating the issue Is this still happening? if so - could you please provide a fresh occurrence (ClientActitvityId )

[cplankl]

@ohadbitt currently I do not execute this part of my application since it is quite risky. The next time when I execute this part I will provide you with a fresh activity id