Push-outputbinding for table using wrong type

[rscottwatson]

Is there anyway to request a field be int64 instead of a int32? I am selecting some azure resources and storing them in a table. When I call Push-OutputBinding -Name OutTable -Value $tableRows for a table that does not exist the function creates the column as INT32 and of course it blows up. Is there anyway to provide some details about the columns that should be created? Or do i need to pre-create the table before loading data to it?

Thanks Scott.

[AnatoliB]

@rscottwatson Are you in control of $tableRows content? What is the type there? Perhaps you could explicitly create values of the type you want. In PowerShell, you can use [long] or [int64] prefixes, like this:

$a = [long]1
$a.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Int64                                    System.ValueType

[rscottwatson]

Thanks @AnatoliB, Yes I am in control of that object. You just made me think of something. Maybe my function app is using 32bit. I will do some more digging otherwise I will try to add the [long] cast.

Thanks. Scott.

[rscottwatson]

I think this might be causing the issue, but at this point it is just a hunch

"use32BitWorkerProcess": true,

From resources.azure.com

[AnatoliB]

@rscottwatson PowerShell in Functions currently always runs in a 32-bit process. We are going to fix this soon to make this match the App platform configuration. However, this may or may not be related to your issue, as it depends very much on how exactly you create this data. For example, $a = 1 will always result in an int32 value, and $a = [long]1 will always give you an int64 value, regardless of the PowerShell process bitness.

[rscottwatson]

So I did further testing and the table that gets created is always created with Int32 regardless if the object that I am passing in my array is a [long] or even an Int64 type.

$tableRows = @()

$graphResults | ForEach-Object {
    write-host "Start process object from azGraph"
    write-host "MaxSizeByte is $($_.properties.readReplicaCount)"
    $_.properties.readReplicaCount.gettype()
    write-host "============================"

    $tableRows += [PSObject]@{
        PartitionKey                  = "Prod"  
        RowKey                        = (new-guid).guid 
         readReplicaCount              = [long]$_.properties.readReplicaCount 
   }
}

write-host "Process PSOBJECT array" 
foreach ( $e in $tableRows ) {   
    write-host "MaxSizeByte from PSOBject is $($e.readReplicaCount)"
    $e.readReplicaCount.gettype()
    write-host "============================"
}
write-host "Done PSOBJECT array" 

Push-OutputBinding -Name AzureInventory -Value $tableRows 

Produced the following output

2019-10-10T13:47:26.424 [Information] INFORMATION: Start process object from azGraph
2019-10-10T13:47:26.588 [Information] INFORMATION: MaxSizeByte is 0
2019-10-10T13:47:26.589 [Information] OUTPUT:
2019-10-10T13:47:26.589 [Information] OUTPUT: IsPublic IsSerial Name                                     BaseType
2019-10-10T13:47:26.589 [Information] OUTPUT: -------- -------- ----                                     --------
2019-10-10T13:47:26.590 [Information] OUTPUT: True     True     Int64                                    System.ValueType
2019-10-10T13:47:26.590 [Information] INFORMATION: ============================
2019-10-10T13:47:26.590 [Information] INFORMATION: Start process object from azGraph
2019-10-10T13:47:26.590 [Information] INFORMATION: MaxSizeByte is 1
2019-10-10T13:47:26.590 [Information] OUTPUT: True     True     Int64                                    System.ValueType
2019-10-10T13:47:26.591 [Information] INFORMATION: ============================
2019-10-10T13:47:26.591 [Information] INFORMATION: Process PSOBJECT array
2019-10-10T13:47:26.591 [Information] INFORMATION: MaxSizeByte from PSOBject is 0
2019-10-10T13:47:26.591 [Information] OUTPUT: True     True     Int64                                    System.ValueType
2019-10-10T13:47:26.597 [Information] INFORMATION: ============================
2019-10-10T13:47:26.597 [Information] INFORMATION: MaxSizeByte from PSOBject is 1
2019-10-10T13:47:26.597 [Information] OUTPUT: True     True     Int64                                    System.ValueType
2019-10-10T13:47:26.597 [Information] INFORMATION: ============================
2019-10-10T13:47:26.597 [Information] INFORMATION: Done PSOBJECT array
2019-10-10T13:47:26.598 [Information] INFORMATION: 10/10/2019 13:47:26
2019-10-10T13:47:26.598 [Information] OUTPUT:

[rscottwatson]

@AnatoliB Tried to make this as simple as possible with the following azure function.

# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)

$tablerow =@{ 
    PartitionKey                  = "TestStorageTable";  
    RowKey                        = (new-guid).guid ;
    smallbit   = 1;
    largebit   = [long]1;
    today      = [datetime](Get-Date)
}

Push-OutputBinding -Name outputTable -Value $tablerow 

# Associate values to output bindings by calling 'Push-OutputBinding'.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
    StatusCode = 200
    Body = "Success"
})

Seems even the datetime column didn't get interpreted correctly.

[AnatoliB]

Thank you @rscottwatson, we'll need to take a closer look.

Could you please clarify this (from your original post):

the function creates the column as INT32 and of course it blows up

What do you mean by that? Do you have something else reading from the table and it breaks because int64 is expected?

I realize this behavior looks surprising, and we will investigate this, but I'm trying to understand which problems this actually causes so that we come up with a reasonable workaround.

Is there anyway to provide some details about the columns that should be created? Or do i need to pre-create the table before loading data to it?

"Columns" in Azure Storage table are not really "columns". The table contains entities (rows), and each entity contains arbitrary properties (except for the three mandatory properties). When property names coincide, GUI tools have an opportunity to visually align them into columns. But properties of each entity instance are still independent of each other, including their data types. As there is no actual concept of "columns", there is no way to pre-create them, but adding entities with different data types in the same properties is technically possible (assuming that the consumer of this data is able to tolerate that).

[rscottwatson]

Hi @AnatoliB

So my original process is reading information about the size of our databases created in azure and I am saving that information to a storage table. One of the columns returned from search-graph is the size of the database in bytes ( which is returned in an int64 variable ) yet the load into the table fails because it tries to save the property as an int32 while push-outputbinding should have selected int64.

Hope that clears things up and thanks for the clarification on a "column" as you can probably guess I am more familiar with SQL databases.

Thanks Scott.

[AnatoliB]

The data is correctly propagated from the PowerShell worker to the Functions host, and then this code in the Storage extension breaks on int64: https://github.com/Azure/azure-webjobs-sdk/blob/ef265e20d4357f2a82871b63a778c99cb6aac8db/src/Microsoft.Azure.WebJobs.Extensions.Storage/Tables/Config/TablesExtensionConfigProvider.cs#L188

        private static EntityProperty CreateEntityPropertyFromJProperty(JProperty property)
        {
            switch (property.Value.Type)
            {
                case JTokenType.String:
                    return EntityProperty.GeneratePropertyForString((string)property.Value);
                case JTokenType.Integer:
                    return EntityProperty.GeneratePropertyForInt((int)property.Value);
                case JTokenType.Boolean:
                    return EntityProperty.GeneratePropertyForBool((bool)property.Value);
                case JTokenType.Guid:
                    return EntityProperty.GeneratePropertyForGuid((Guid)property.Value);
                case JTokenType.Float:
                    return EntityProperty.GeneratePropertyForDouble((double)property.Value);
                default:
                    return EntityProperty.CreateEntityPropertyFromObject((object)property.Value);
            }
        }

There are at least two issues here:

• This code tries to convert any JTokenType.Integer into int. When the value is small enough to fit int, it is propagated further as int, and the resulting Azure table entity property type becomes int32. However, when the value is too large to fit into int, this code throws an exception and no entity is added to the Azure table.
• There is no code branch that would return DateTime. The value comes as JTokenType.String, so this property ends up being added to the resulting Azure table entity as a string.

Fixing this would require changes in the extension code, and most likely changes int the language worker communication protocol.

This should be affecting all out-of-proc languages. In-proc C# does not seem to be affected. We'll need to think about how, when, and whether make a change here, as this change would be potentially breaking.

The workaround in the meantime: instead of using the output binding for storage table, use the Azure SDK directly or find/implement PS modules that can handle this better.

[rscottwatson]

Thanks @AnatoliB I am looking into using the AzTable module to see if that will provide different results.

[AnatoliB]

One more workaround idea: output these values as strings using any serialization you can (Json, .ToString(), etc.), and make sure the consumer of this data knows how to deserialize it.