Parameter data from the helpscraper can be too big to fit in Azure table property

[Jawz84]

It seems there there are two problems here: one is that for some cmdlets, the parameter help is so big, it doesn't fit the 64k limit in an Azure Table storage property. This is why Import-Module -name didn't return any help (-allowclobber is not a valid param here). So that part needs some work, because it means some module help imports may have failed because the Parameter property data was too big. It appears I can compress the json before uploading, and that saves quite a bit of space. This is what I did to fix this for Import-Module. But now I have to figure out a way to fix this at scale.

The other problem (where the parameter data does not fit the schema) I have not even touched yet.

Originally posted by @Jawz84 in https://github.com/Jawz84/explainpowershell/issues/64#issuecomment-1133590975

[Jawz84]

TODO:

• think of more ways to further compress the json-parameterdata (compressing by removing redundant parts or default values?)
• alternatively, think of a different way to store parameterdata (maybe in a separate table or in a separate Partition in Azure table storage?
• rerun help-scraping on all modules, but this time with the new, compressed way of storing parameterdata.

[Jawz84]

I have updated the helpscraper to at least do a -Compress on the json. Which already helps a lot.

TODO:

• find all modules in the DB with cmdlets that have Parameter that is empty, and rerun them trough the helpscraper

$ct = (Get-AzStorageTable -name helpdata -Context $ctx ).CloudTable
$data = $data ?? (Get-AzTableRow -Table $ct -PartitionKey CommandHelp)

$entities = $data 
    | Where-Object { ($_.parameters -eq $null -and $_.Syntax -match "\[" -and $_.Syntax.trim() -ne "$($_.CommandName) [<CommonParameters>]")} 
    | Select-Object ModuleName, CommandName| Group-Object ModuleName

[Jawz84]

After fixing lots of modules, these four cmdlets remain:

$ctx = Get-AzStorageAccount -Name explainpowershell -ResourceGroupName powershellexplainer | Select-Object -ExpandProperty Context
$ct = (Get-AzStorageTable -name helpdata -Context $ctx ).CloudTable
$data = $data ?? (Get-AzTableRow -Table $ct -PartitionKey CommandHelp)

$entities = $data
    | Where-Object { ($_.parameters -eq $null -and $_.Syntax -match "\[" -and $_.Syntax.trim() -ne "$($_.CommandName) [<CommonParameters>]")} 
    | Select-Object ModuleName, CommandName| Group-Object ModuleName

$entities

Count Name                      Group
----- ----                      -----
    2 Az.ServiceFabric          {@{ModuleName=Az.ServiceFabric; CommandName=Add-AzServiceFabricClusterCertificate}, @{ModuleName=Az.ServiceFabric; CommandName=Remove-AzServiceFabricClusterCertificate}}
    1 Az.StorageSync            {@{ModuleName=Az.StorageSync; CommandName=Invoke-AzStorageSyncFileRecall}}
    1 PackageManagement         {@{ModuleName=PackageManagement; CommandName=Install-Package}}

[Jawz84]

And now, all of these have been resolved. Closing as fixed.

[Jawz84]

related to #63