f5devcentral / f5-azure-saca

Secure Azure Computing Architecture for DoD (SACA) - Notional Secure Cloud Computing Architecture (SCCA) Deployment
MIT License
26 stars 39 forks source link

Deployment in Commercial and Gov Fails #90

Closed Mikej81 closed 4 years ago

Mikej81 commented 4 years ago

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"Conflict","message":"{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'start'. Error message: \\"Enable failed: failed to execute command: command terminated with exit status=1\n[stdout]\nabout to execute\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nloading verifyHash script\ncannot validate signature of /config/verifyHash\n\n[stderr]\nData Input Error: The path \\"/config/verifyHash\\" is invalid. Check the valid paths with tmsh list sys global-settings file-whitelist-path-prefix file-blacklist-path-prefix file-blacklist-read-only-path-prefix\n/bin/sh: line 36: /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/util.sh: No such file or directory\n\\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot \"\r\n }\r\n ]\r\n }\r\n}"},{"code":"Conflict","message":"{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'start'. Error message: \\"Enable failed: failed to execute command: command terminated with exit status=1\n[stdout]\nabout to execute\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nloading verifyHash script\ncannot validate signature of /config/verifyHash\n\n[stderr]\nData Input Error: The path \\"/config/verifyHash\\" is invalid. Check the valid paths with tmsh list sys global-settings file-whitelist-path-prefix file-blacklist-path-prefix file-blacklist-read-only-path-prefix\n/bin/sh: line 36: /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/util.sh: No such file or directory\n\\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot \"\r\n }\r\n ]\r\n }\r\n}"}]}

Mikej81 commented 4 years ago

Serial Log From Azure.

f5dnst1-f5vmt10.0bc1d2b7-c738-45b7-b61f-682425c355b3.serialconsole.txt

PlateSpinner commented 4 years ago

I don't have the serial console output but I can confirm this is happening on Azure Gov with 3NIC_3Tier and 3NIC_1Tier deployments. Same error on the VM Extension.

It appears to be triggered when the extension runs the commands in the "installCloudLibs" ARM variable. Specifically usr/bin/tmsh load sys config merge file /config/verifyHash

So either the "verifyHash" file isn't created by the command echo -e ', variables('verifyHash64'), ' | base64 -d > /config/verifyHash or maybe this is a new "file-whitelist-path-prefix" thing? (like this was)

milazzom commented 4 years ago

I believe I wrote up a fix in my own fork of this repo late last year... I only modified the 3-tier 3NIC HA template because that's what we were using, but I'll look into updating it for the single tier template as well.

Template