ArcGIS Enterprise 11.1: Portal federation failed with invalid token

Biboba commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request

Module Version

4.1.0

Affected Resource(s)

ArcGIS_Federation

Configuration Files

GisPortalAGSDeployment_anonymous.txt

Expected Behavior

Testing if ArcGIS Server is federated or not should work.

Actual Behavior

Testing if ArcGIS Server is federated fails with the following error:

{"error":{"code":498,"message":"Invalid Token.","details":["Token would have expired, regenerate token and send the request again.","If the token is generated based on the referrer make sure the referrer information is available with every request in header."]

Steps to Reproduce

Run the attached configuration to federate the ArcGIS Server.

Important Factoids

ArcGIS Enterprise 11.1 is not a fresh installation. It's from an upgrade from 11.0 in case that could have an influence.
Federation test works fine on an ArcGIS Enterprise 11.0 deployment but not on 11.1 one.
If I generate the token for a referer without specifying the port 443, it works. I wonder if it is an ArcGIS Enterprise bug (Portal for ArcGIS or WebAdaptor ?) or a DSC one. Please let me know @cameronkroeker.

In ArcGIS_Federation.psm1:
```
$Referer = "https://$($PortalFQDN):$($PortalPort)/$PortalContext"
$token = Get-PortalToken -PortalHostName $PortalFQDN -Port $PortalPort -SiteName $PortalContext -Credential $RemoteSiteAdministrator -Referer $Referer
```

instead use:

$Referer = "https://$($PortalFQDN)/$PortalContext"

and it works.

Can you reproduce on your side ?

Thanks !

References

cameronkroeker commented 1 year ago

Hi @Biboba,

Is IWA enabled on the portal WA? Also, does it work if you change the following three attributes to use the local portal hostname, port 7443, and arcgis context? For example:

"PortalHostName": "interla-portal.domain.com",
"PortalPort": "7443",
"PortalContext": "arcgis",

Thanks, Cameron K.

Biboba commented 1 year ago

Hi @cameronkroeker,

Thanks for your reply.

IWA is not enabled on the portal WA
If I update port and context to bypass Portal for ArcGIS webadaptor as suggested, it does work.

Can you reproduce the issue on your side if webadaptor context is specified ?

Thanks

cameronkroeker commented 1 year ago

Thanks @Biboba. I have reproduced this with both the module, and outside the module using Postman. It's interesting that its only with 11.1, as it works fine with 11.0 and below. So perhaps there was a change in the token validation at 11.1 that I am not aware of.

Biboba commented 1 year ago

Thanks @cameronkroeker for the reply.

I wonder if it is rather not a WebAdaptor for IIS bug. Can you eventually reproduce on a Java one ?
If it works when bypassing the webadaptor and contacting the Portal for ArcGIS directly, to me it means it's a WA bug.

Please let me know if I can help.

Thanks

cameronkroeker commented 1 year ago

Hi @Biboba and @dimesv,

I am still not sure why this doesn't work in 11.1 but works in 11.0 and below, however, below is another workaround (originally suggested by @dimesv!) that can be used, and will be the fix that will be included in the next release of the module:

After line 137 https://github.com/Esri/arcgis-powershell-dsc/blob/8621f7a7256d6f8b9431caf4674e01c8d3bce249/Modules/ArcGIS/DSCResources/ArcGIS_Federation/ArcGIS_Federation.psm1#L137 add the following snippet:

if($PortalPort -eq 443){
   $Referer = "https://$($PortalFQDN)/$PortalContext"
}

Then after line 337 https://github.com/Esri/arcgis-powershell-dsc/blob/8621f7a7256d6f8b9431caf4674e01c8d3bce249/Modules/ArcGIS/DSCResources/ArcGIS_Federation/ArcGIS_Federation.psm1#L337 add the following snippet:

if($PortalPort -eq 443){
   $Referer = "https://$($PortalFQDN)/$PortalContext"
}

Thanks, Cameron K.

Biboba commented 1 year ago

Hi @cameronkroeker,

Thanks for the reply and the fix. Out of curiosity, did you get the chance to test this behavior on ArcGIS WebAdaptor for Java ? If this a bug of the IIS version, might be worth reporting it as other users may face it in other situations than 'arcgis-powershell-dsc'.

Thanks !

cameronkroeker commented 1 year ago

Hi @cameronkroeker,

Thanks for the reply and the fix. Out of curiosity, did you get the chance to test this behavior on ArcGIS WebAdaptor for Java ? If this a bug of the IIS version, might be worth reporting it as other users may face it in other situations than 'arcgis-powershell-dsc'.

Thanks !

Hi @Biboba,

I haven't had a chance to test this behavior with the Java Web Adaptor, however in my original effort I was able to replicate it outside of the PowerShell DSC ArcGIS Module via PostMan. I've reported the issue and it is being investigated.

Thanks, Cameron K.

cameronkroeker commented 1 year ago

Hi @Biboba,

I was able to confirm this is a bug specific to the IIS 11.1 Web Adaptor, which will be addressed in an upcoming patch. Once the patch goes live I will post it here.

Thanks, Cameron K.

Biboba commented 1 year ago

Hi @cameronkroeker,

Thanks for the update and the great follow up! Happy to read this behavior has been understood and identified as a bug. Will remove the workaround once the patch will be live.

Thanks, Nicolas

Biboba commented 1 year ago

Hi @cameronkroeker,

I deployed the patch released fixing this issue: https://support.esri.com/en-us/patches-updates/2023/arcgis-web-adaptor-iis-11-1-reliability-patch

Unfortunately, it keeps being reinstalled whenever the configuration is run:

10.07.2023 17:21:44: [PORTAL01]:                            [[ArcGIS_InstallPatch]WebAdaptorInstallPatch] Checking Patch File at \\PATHTO\setup\patches\11.1\webadaptor\ArcGIS-111-WAI-R-Patch.msp
10.07.2023 17:21:49: [PORTAL01]:                            [[ArcGIS_InstallPatch]WebAdaptorInstallPatch] Patch File at \\PATHTO\setup\patches\11.1\webadaptor\ArcGIS-111-WAI-R-Patch.msp with QFE Id WAI-111-P-942 not installed

Though it seems, this time the patch has proper metadata:

Could it be an "arcgis-powershell-dsc" issue this time ?

Thanks,

Nicolas

cameronkroeker commented 1 year ago

Hi @Biboba,

This is happening because the QFE_ID within the Windows registry does not match the QFE_ID extracted from the patch (.msp) file or the json website:

MicrosoftTeams-image

DSC uses these values to determine if the patch needs to be installed or not. If they match then install is skipped.

I have reported this to the patch team.

Thanks, Cameron K.

Biboba commented 1 year ago

Hi @cameronkroeker,

Thanks for quick reply and for the report to the patch team: much appreciated. Could you please update this thread once new patch is available ?

Thanks !

mjperez-usgs commented 1 year ago

Hi @Biboba and @dimesv,

I am still not sure why this doesn't work in 11.1 but works in 11.0 and below, however, below is another workaround (originally suggested by @dimesv!) that can be used, and will be the fix that will be included in the next release of the module:

After line 137

https://github.com/Esri/arcgis-powershell-dsc/blob/8621f7a7256d6f8b9431caf4674e01c8d3bce249/Modules/ArcGIS/DSCResources/ArcGIS_Federation/ArcGIS_Federation.psm1#L137

add the following snippet:
if($PortalPort -eq 443){
   $Referer = "https://$($PortalFQDN)/$PortalContext"
}
Then after line 337

https://github.com/Esri/arcgis-powershell-dsc/blob/8621f7a7256d6f8b9431caf4674e01c8d3bce249/Modules/ArcGIS/DSCResources/ArcGIS_Federation/ArcGIS_Federation.psm1#L337

add the following snippet:
if($PortalPort -eq 443){
   $Referer = "https://$($PortalFQDN)/$PortalContext"
}
Thanks, Cameron K.

I've tried to edit the powershell module, installed to , with this fix - however, nothing ever changes. I even add some Write-Verbose statements, but they never output. Is the file to edit the one in C:\Program Files\WindowsPowerShell\Modules\ArcGIS\4.1.0\DSCResources\ArcGIS_Federation on the system you are running Invoke-ArcGIS-Configuration from? I've made sure to close the terminal and reload the module. Since the webadaptor patch doesn't seem to work (it won't even install for me) and this workaround is not working, I'm kind of dead in the water right now waiting for a fix.

I'm not really versed in developing powershell modules so I'm not sure how to go about making this change or running a version from the repo.

rlhadsel commented 1 year ago

Hey @mjperez-usgs, that change will need to be made to all of the machines (including the orchestration machine) participating in the deployment. Ensure the change is made to all of the machines, close out all PowerShell windows, re-open PowerShell as an Administrator and retry the Invoke-ArcGISConfiguration command again.

mjperez-usgs commented 11 months ago

Is there any timeline on when this will be fixed in a release? While this is a simple fix to do by hand, when put into an automatically scaling setup it is a significant amount of additional work to add.

cameronkroeker commented 10 months ago

Is there any timeline on when this will be fixed in a release? While this is a simple fix to do by hand, when put into an automatically scaling setup it is a significant amount of additional work to add.

Yes, the fix will be included in the next release of the ArcGIS Module, v4.2.0, which is set to be released alongside ArcGIS Enterprise 11.2.

Thanks, Cameron K.

cameronkroeker commented 10 months ago

This issue has been addressed in release v4.2.0:

https://github.com/Esri/arcgis-powershell-dsc/releases/tag/v4.2.0

Esri / arcgis-powershell-dsc