JulianHayward / Azure-MG-Sub-Governance-Reporting

Azure Governance Visualizer aka AzGovViz is a PowerShell script that captures Azure Governance related information such as Azure Policy, RBAC (a lot more) by polling Azure ARM, Storage and Microsoft Graph APIs.
MIT License
814 stars 293 forks source link

error: The type eq 'subscription' or type eq 'managementgroup' filter is invalid. #217

Open patkiamit opened 7 months ago

patkiamit commented 7 months ago

AzGovViz version 6.3.4

CodeRunPlatform Azure DevOps,

Describe the bug azure DevOps pipeline failed with the below error

Screenshots 2023-11-29T11:46:00.5004337Z !f97434b8 Please report at aka.ms/AzGovViz and provide the following dump 2023-11-29T11:46:00.5014629Z [AzAPICallErrorHandler 1.1.84] Get PIM onboarded Subscriptions and Management Groups try #1; uri:"https://graph.microsoft.com/beta/privilegedAccess/azureResources/resources?$select=id%2cdisplayName%2ctype%2cexternalId&$expand=parent&$filter=(type+eq+%27subscription%27+or+type+eq+%27managementgroup%27)&$skiptoken=b4NY2ZLEyNFBEHQKBIVA"; return: (StatusCode: '400' (BadRequest)) <.code: ''> <.error.code: 'InvalidFilter'> | <.message: ''> <.error.message: 'The type eq 'subscription' or type eq 'managementgroup' filter is invalid.'> - (plain : @{error=}) - AzAPICall: Stop 2023-11-29T11:46:00.5019099Z Parameters: 2023-11-29T11:46:00.5026304Z accountType:ServicePrincipal 2023-11-29T11:46:00.5049562Z ARMLocations:asia asiapacific australia australiacentral australiacentral2 australiaeast australiasoutheast brazil brazilsouth brazilsoutheast brazilus canada canadacentral canadaeast centralindia centralus centraluseuap centralusstage eastasia eastasiastage eastus eastus2 eastus2euap eastus2stage eastusstage eastusstg europe france francecentral francesouth germany germanynorth germanywestcentral global india israelcentral italynorth japan japaneast japanwest jioindiacentral jioindiawest korea koreacentral koreasouth northcentralus northcentralusstage northeurope norway norwayeast norwaywest polandcentral qatarcentral singapore southafrica southafricanorth southafricawest southcentralus southcentralusstage southcentralusstg southeastasia southeastasiastage southindia sweden swedencentral switzerland switzerlandnorth switzerlandwest uae uaecentral uaenorth uk uksouth ukwest unitedstates unitedstateseuap westcentralus westeurope westindia westus westus2 westus2stage westus3 westusstage 2023-11-29T11:46:00.5059455Z azAccountsVersion:n/a 2023-11-29T11:46:00.5069080Z azAPICallModuleVersion:1.1.84 2023-11-29T11:46:00.5078287Z azureCloudEnvironment:AzureCloud 2023-11-29T11:46:00.5087616Z codeRunPlatform:AzureDevOps 2023-11-29T11:46:00.5097099Z debugAzAPICall:False 2023-11-29T11:46:00.5116015Z debugWriteMethod:Host 2023-11-29T11:46:00.5124248Z DoAzureConsumption:False 2023-11-29T11:46:00.5133744Z DoNotIncludeResourceGroupsAndResourcesOnRBAC:False 2023-11-29T11:46:00.5143333Z DoNotIncludeResourceGroupsOnPolicy:False 2023-11-29T11:46:00.5152765Z DoNotShowRoleAssignmentsUserData:True 2023-11-29T11:46:00.5406294Z DoPSRule:False 2023-11-29T11:46:00.5418259Z GitHubActionsOIDC:False 2023-11-29T11:46:00.5427382Z gitHubRepository:aka.ms/AzGovViz 2023-11-29T11:46:00.5436805Z HierarchyMapOnly:False 2023-11-29T11:46:00.5447107Z LargeTenant:True 2023-11-29T11:46:00.5458207Z ManagementGroupsOnly:False 2023-11-29T11:46:00.5467439Z NoALZPolicyVersionChecker:False 2023-11-29T11:46:00.5476917Z NoJsonExport:False 2023-11-29T11:46:00.5486003Z NoMDfCSecureScore:True 2023-11-29T11:46:00.5496596Z NoNetwork:False 2023-11-29T11:46:00.5507005Z NoPolicyComplianceStates:False 2023-11-29T11:46:00.5516080Z NoResourceProvidersAtAll:True 2023-11-29T11:46:00.5525011Z NoResourceProvidersDetailed:False 2023-11-29T11:46:00.5534091Z NoResources:False 2023-11-29T11:46:00.5543454Z NoStorageAccountAccessAnalysis:False 2023-11-29T11:46:00.5561542Z onAzureDevOps:True 2023-11-29T11:46:00.5569978Z onAzureDevOpsOrGitHubActions:True 2023-11-29T11:46:00.5586796Z onGitHubActions:False 2023-11-29T11:46:00.5595841Z PolicyAtScopeOnly:True 2023-11-29T11:46:00.5614891Z ProductVersion:6.3.4 2023-11-29T11:46:00.5631170Z PSRuleFailedOnly:False 2023-11-29T11:46:00.5649071Z psVersion:7.2.16 2023-11-29T11:46:00.5656990Z RBACAtScopeOnly:True 2023-11-29T11:46:00.5675034Z skipAzContextSubscriptionValidation:False 2023-11-29T11:46:00.5692046Z subscriptionId4AzContext:undefined 2023-11-29T11:46:00.5708654Z subscriptionQuotaId:EnterpriseAgreement_2014-09-01 2023-11-29T11:46:00.5726085Z tenantId4AzContext:undefined 2023-11-29T11:46:00.5734626Z ThrottleLimit:10 2023-11-29T11:46:00.5743830Z userType:n/a 2023-11-29T11:46:00.5752921Z writeMethod:Host 2023-11-29T11:46:00.5759010Z [AzAPICallErrorHandler 1.1.84] Get PIM onboarded Subscriptions and Management Groups try #1; uri:"https://graph.microsoft.com/beta/privilegedAccess/azureResources/resources?$select=id%2cdisplayName%2ctype%2cexternalId&$expand=parent&$filter=(type+eq+%27subscription%27+or+type+eq+%27managementgroup%27)&$skiptoken=b4NY2ZLEyNFBEHQKBIVA"; return: (StatusCode: '400' (BadRequest)) <.code: ''> <.error.code: 'InvalidFilter'> | <.message: ''> <.error.message: 'The type eq 'subscription' or type eq 'managementgroup' filter is invalid.'> - unhandledErrorAction: Stop 2023-11-29T11:46:00.6338041Z Exception: 2023-11-29T11:46:00.6338477Z Line | 2023-11-29T11:46:00.6339082Z  832 |  Throw 'Error - check the last console output for details' 2023-11-29T11:46:00.6340173Z  |  ~~~~~~~~~~~~~ 2023-11-29T11:46:00.6340986Z  | Error - check the last console output for details 2023-11-29T11:46:00.7383355Z ##[error]PowerShell exited with code '1'. 2023-11-29T11:46:00.7434406Z ##[section]Finishing: Run Azure Governance Visualizer

Additional context Add any other context about the problem here.

JulianHayward commented 7 months ago

@patkiamit thanks for reporting, but it sounds odd :) Does this happen every time you run it?

What happens if you run this?

$uri = 'https://graph.microsoft.com/beta/privilegedAccess/azureResources/resources?$select=id%2cdisplayName%2ctype%2cexternalId&$expand=parent&$filter=(type+eq+%27subscription%27+or+type+eq+%27managementgroup%27)'

invoke-azrestmethod -uri $uri

ref - connect as service principal

patkiamit commented 7 months ago

Hi, thanks for the reply, I tested above URL and was able to get the output , still 28th NOV all work well no issue it started from y'day

Headers : {[Transfer-Encoding, System.String[]], [Strict-Transport-Security, System.String[]], [request-id, System.String[]], [client-request-id, System.String[]]...} Version : 1.1 StatusCode : 200 Method : GET Content : {"@odata.context":"https://graph.microsoft.com/beta/$metadata#governanceResources(id,displayName,type,externalId,parent())","@odata.nextLink":"https://graph.microsoft.com/beta/privilegedAccess/az ureResources/resources?$select=id%2cdisplayName%2ctype%2cexternalId&$expand=parent&$filter=(type+eq+%27subscription%27+or+type+eq+%27managementgroup%27)&$skiptoken=L0DwjWpAm0udxnBgvv-uCQ","value" :[............................................................................................................long output.......................................................... teTime":null,"managedAt":null,"registeredRoot":null,"originTenantId":null}}]}

kaiaschulz commented 7 months ago

@patkiamit I was able to reproduce the issue on my side as well. First call was successful (http status code: 200 (OK)) and all other are failing (http status code: 400 (BadRequest)), which is related to the $skiptoken.

patkiamit commented 7 months ago

@patkiamit I was able to reproduce the issue on my side as well. First call was successful (http status code: 200 (OK)) and all other are failing (http status code: 400 (BadRequest)), which is related to the $skiptoken.

i run URL separately with skiptoken as well, and it runs without issue , but in script, it always fails

kaiaschulz commented 7 months ago

Hi @patkiamit, this Microsoft Graph API request is responding with 200 results by default. So, the problem should only happens if your count is greater than that. Even with a $top=999 it will max. respond with 200 results. If you have more than the default 200 PIM assignments (after 28th November?), this will cause the issue. After that, it is starting the paging. It seems that the $filter is somehow the problem of the second call in combination with the $skiptoken.

How were you able to use the url with skiptoken? Could you please provide your test?

In my case, AzAPICall nor Invoke-AzRestMethod-command is working.

OUTPUT:

{"error":{"code":"InvalidFilter","message":"The type eq 'subscription' or type eq 'managementgroup' filter is invalid.","innerError":{"date":"2023-11-30T11:12:46","request-id":"x,"client-request-id":"y"}}}

Nevertheless, without $filter it isn't working as well:

{"error":{"code":"InvalidFilter","message":"The  filter is invalid.","innerError":{"date":"2023-11-30T11:26:57","request-id":"xx","client-request-id":"yy"}}}
patkiamit commented 7 months ago

Hi @patkiamit, this Microsoft Graph API request is responding with 200 results by default. So, the problem should only happens if your count is greater than that. Even with a $top=999 it will max. respond with 200 results. If you have more than the default 200 PIM assignments (after 28th November?), this will cause the issue. After that, it is starting the paging. It seems that the $filter is somehow the problem of the second call in combination with the $skiptoken.

How were you able to use the url with skiptoken? Could you please provide your test?

In my case, AzAPICall nor Invoke-AzRestMethod-command is working.

OUTPUT:

{"error":{"code":"InvalidFilter","message":"The type eq 'subscription' or type eq 'managementgroup' filter is invalid.","innerError":{"date":"2023-11-30T11:12:46","request-id":"x,"client-request-id":"y"}}}

Nevertheless, without $filter it isn't working as well:

{"error":{"code":"InvalidFilter","message":"The  filter is invalid.","innerError":{"date":"2023-11-30T11:26:57","request-id":"xx","client-request-id":"yy"}}}

yes, I am running a script in a large tenant having many subscriptions, do we have any workaround for it?

JulianHayward commented 7 months ago

meanwhile seeing/hearing this from other tenants, too.

Workaround until the issue is fixed: use the parameter -NoPIMEligibility.

Using a beta Microsoft Graph API here which is announced for deprecation; elaboration migration path to Azure Resource Manager (ARM) API

JulianHayward commented 6 months ago

reopen for reference / to be closed when fixed

kaiaschulz commented 3 months ago

Hey @JulianHayward, any updates on this topic? The eligible assignments of the Privileged Identity Management (PIM) are giving us still a hard time. Actually tested with v6.4.3.

JulianHayward commented 3 months ago

hey @kaiaschulz - no progress, yet :(

stevenoneill commented 1 month ago

Love the tool! We're also hitting this. Keeping an eye on the thread.