search

Esri / arcgis-azure-templates

Apache License 2.0
29 stars 14 forks source link

Deploying in locked-down environment. What needs to be whitelisted for successful deployment? #8

Open PapachoSneak opened 3 years ago

PapachoSneak commented 3 years ago

We have had success deploying across several environments and are close to deploying in our Production environment. Unfortunately, we're being told that the virtual network we'll be deploying to is locked down with no internet access by default, though we can ask for explicit exceptions. We've tried to run our deployment automation (based on the ArcGIS Azure templates here) in this new locked-down environment and it's failing on the step where it configures the fileshare VM. We get the following error:

"VM has reported a failure when processing extension 'cse'. Error message: \"Failed to download all specified files. Exiting. Error message: Unable to connect to the remote server..."

We think this might be related to the fact that the fileshare VM cannot see the internet, but aren't sure from the error message exactly what it's trying to access if so. A few questions:

Is there anyplace we can look to get a more detailed log of what it's failing on?

We searched through all of the deployment scripts in the DSC.zip file the deployment uses to try and find any intenet URLs referenced in them and found 3:

The $GitOpenSSL32BitInstaller one is in the FileShareConfiguration.ps1 file, and seems a likely candidate since that's the step our deployment is failing on with the "Unable to connect to the remote server" message, Could this be what the message is telling us the automation can't connect to?

Is there a list of all of the internet URLs we will need to have whitelisted for any or all of the ESRI automation templates to work?

image

PapachoSneak commented 3 years ago

Any guidance here? Thanks!

bailsman commented 3 years ago

(just another user, not representing Esri)

@PapachoSneak

You may be interested in the log files that are located at C:\WindowsAzure\Logs\Plugins\Microsoft.Powershell.DSC\ See also: https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/dsc-overview There should be a file there that can tell you exactly which task has failed and that should make it easy for you to figure out the next URL that needs to be whitelisted.

As to finding them all in one shot, just searching the code base for urls - as you have already done - seems to be a sensible approach. I don't know that anyone maintains a full list of them.

PapachoSneak commented 3 years ago

Hey @bailsman -

Thanks for the response, and the info on logging - we will be checking those out for sure.

We were able to get temporary approval for internet access in our new environment and confirmed that the issue we had seen was solved once the VMs could see the internet. Just not 100% sure which URL(s) were used. Will dig through the logs to see what they say.

About the URL whitelist - @shailesh91 - any insight would be much appreciated!

Thanks

PapachoSneak commented 3 years ago

Still waiting on a response on this - @shailesh91 can you help? Also, we just got word that we might not be able to have any internet access at all during deployment. We're going to try to push forward with getting whitelist exceptions, but if we can't, do we have a path forward for using these templates? I can imagine copying the relevant files the scripts pull down from the internet and hosting them internally somehwere... But I know the automations validate the software license(s) with ESRI and bail out if they're not valid. If the VMs can't see ESRI to validate those, is there a workaround?

Thank you for any help here.

shailesh91 commented 3 years ago

@PapachoSneak your error is directly related to being not able to access internet, specifically the deployment artifacts storage account, that is used to stage the artifacts. Looks like your, the custom script extension is trying to download the GenerateSSLCerts.ps1 from the storage account and is not able. You will run into similar errors, even if you get past this as the DSC extension will try to access the same storage account. Currently there is no alternative, other trying to whitelisting the deployment artifacts storage account endpoint.

PapachoSneak commented 3 years ago

@shailesh91 - thanks for the response. We'll keep the deployment artifacts storage account endpoint on our list for whitelisting. But what else needs to be whitelisted? Our IA folks are telling us they require a list of exactly what we need (what these ArcGIS Azure templates require). As noted in the original posting for this issue, we found 3 URLs in the DSC.zip file. None of those are for esri.com, which I would imagine would need to be in the whitelist for the software authorization step?

It really all boils down to the question:

Is there a list of all of the internet URLs we will need to have whitelisted for any or all of the ESRI ArcGIS Azure templates to work?

shailesh91 commented 2 years ago

@PapachoSneak unfortunately, we don't have a definite list of URLs that are required to be whitelisted. But we are willing to provide some starting cues to build a list and test it out. Some of the endpoints that need to be definitely whitelisted are

  1. Deployment storage account ([deployment-storage-account-name].blob.core.windows.net)
  2. Licensing server on esri.com
  3. arcgis.com

Test whitelisting these URLs and if you run into any failures let us know.

PapachoSneak commented 2 years ago

Thanks for the response, @shailesh91 . We will do as advised and let you know if we see failures.