[Feature] Add Data Processing Azure Function

[briandelmsft]

Logic apps have some limitations that limit data processing functionality. For example, sorting an array of JSON objects by a property of the JSON object. Also, dealing with complex API output like that of the Log Analytics API require Logic apps to go through many iterations of a loop which makes dealing with large datasets difficult and slower.

Add an Azure Function which can be called behind the scenes by STAT modules to aide in these data processing functions

[piaudonn]

I Am So Excited About This One :)

[piaudonn]

If we customize the function name, the deployment might fail if the provided name is using a forbidden character for the storage account or a name too long once we concatenate the name and the random string. Would that deserve a check or a string operation in the name generation?

[briandelmsft]

If we customize the function name, the deployment might fail if the provided name is using a forbidden character for the storage account or a name too long once we concatenate the name and the random string. Would that deserve a check or a string operation in the name generation?

Yeah, right now there's a lowercase function on it but that is all. We can do regex validation on the text box input.

https://learn.microsoft.com/en-us/azure/azure-resource-manager/managed-applications/microsoft-common-textbox

[briandelmsft]

Validation has been added so that you can't pass an invalid name/length for the function name

I also added validations to ensure permissions to create logic apps, Azure Functions, App Service Plans and Storage accounts are in place

[briandelmsft]

@piaudonn

I've got the function now working in PowerShell which should give greater region support. I'm having a challenge with the deployment though. The function consistently deploys without error and seems to work fine, but 1-2 of 3 modules (Base, Scoring and KQL) are now inconsistently failing to deploy. Consistently at least 1 fails, usually 2. Now those modules are the only ones that reference the function, but they are all dependent on the function deployment in the template so it shouldn't be an issue. The stranger part, if you run the deployment a second time it all works fine.

Deploy PS Function

{
    "status": "Failed",
    "error": {
        "code": "BadRequest",
        "message": "{\"Code\":\"BadRequest\",\"Message\":\"Encountered an error (InternalServerError) from host runtime.\",\"Target\":null,\"Details\":[{\"Message\":\"Encountered an error (InternalServerError) from host runtime.\"},{\"Code\":\"BadRequest\"},{\"ErrorEntity\":{\"Code\":\"BadRequest\",\"Message\":\"Encountered an error (InternalServerError) from host runtime.\"}}],\"Innererror\":null}"
    }
}

[piaudonn]

@briandelmsft I think this one can be closed as the data processing is inherently a part of statv2 function.