Unable to run in "unsafe mode"

[jasonb815]

Type of issue

• [x ] Bug
• [ ] New feature
• [ ] Enhancement

Description

Receiving the following error when I try to deploy an older PCS version (1.0.2) via CLI

pcs -s basic --versionOverride 1.0.2 --dockerTagOverride 1.0.2

✕ Deployment failed { "status": "Failed", "error": { "code": "ResourceDeploymentFailure", "message": "The resource operation completed with terminal provisioning state 'Failed'.", "details": [ { "code": "VMExtensionProvisioningError", "message": "VM has reported a failure when processing extension 'scriptextensions'. Error message: \"Enable failed: processing file downloads failed: failed to download file[1]: failed to download file: unexpected status code: got=404 expected=200\"." } ] } }

...

Steps to reproduce

1. pcs -s basic --versionOverride 1.0.2 --dockerTagOverride 1.0.2

Expected behavior

I expect the command to create a Remote Monitoring PCS at version 1.0.2

Current behavior <!-- The actual behavior observed --

Error above

Known workarounds

...

Possible solution

...

Context and Environment

• Operating System: ...
• GitHub branch: ...
• Node.js Runtime: ...

[ppathan]

@jasonb815 Any reason for using the 1.0.2 version. The latest version is 2.1.2 which was release last month.

[jasonb815]

An app we are working on interfaces with the PCS and requires PCS to be started in "UNSAFE" mode. UNSAFE mode is not working in 2.1.2 as best as we can tell as we get 502 errors when trying to connect and the PCS UI goes dormant until we restart services back in SAFE mode. Make sense? All of this works fine in 1.0.2 and has worked there for months. There are no "new" features as far as I know in the 2.x release that we need at this time.

[ppathan]

What command are you using?

This command should work:

cd /app
sudo ./start.sh --unsafe

Is it not working for you?

[jasonb815]

yup.. it starts in unsafe mode just fine and throws no errors...

[ppathan]

Can this issue be closed?

[jasonb815]

NO!

[jasonb815]

We need to be able to deploy 1.0.2 in order for UNSAFE mode to work for us.

[ppathan]

I'm confused. I asked if the above command works for you and i thought you meant yes

[jasonb815]

Let me explain..

We are able to start BOTH 1.0.2 and 2.1.2 PCS in UNSAFE mode with the above command.

We are ABLE to connect to 1.0.2 PCS in UNSAFE mode via our application

We are UNABLE to connect to 2.1.2 PCS in UNSAFE mode via our application (502 error).

Nothing else has changed.

[ppathan]

@jillcary Can you please take a look?

[jillcary]

@jasonb815 how did you install pcs-cli? From a branch (master or 2.1.2 etc...) or via npm (npm install -g iot-solutions)?

[jasonb815]

checked it out from GIT Hub and then did the following...

- pcs login
- npm install
- npm start
- npm link
- pcs -s basic --versionOverride 1.0.2 --dockerTagOverride 1.0.2

[jillcary]

@jasonb815 can you confirm what version is printed when you type pcs -v?

[jasonb815]

JasonB-MBP-2018:publish jason$ pcs -v 2.1.2

[jillcary]

Can you also provide the output from your VM when you type docker ps -a? this will show all of the running docker containers.

Edit: can you do this on your 2.1.2 deployment

[jasonb815]

I'm not on a VM.. I'm on my local machine...

[jillcary]

Ok, you should also be able to run the command locally to check your running containers. Can you also check your local machine's env vars and make sure the PCS_AUTH_REQUIRED environment variable is set to "false".

It is possible that the start script locally is executing the export PCS_AUTH_REQUIRED="false" command but it isn't being set globally.

[jillcary]

We have more information on global env vars for your system here if needed.

[jasonb815]

local ENV 👍 JasonB-MBP-2018:publish jason$ ENV TERM_PROGRAM=Apple_Terminal SHELL=/bin/bash TERM=xterm-256color TMPDIR=/var/folders/rf/jdxqm_kn7ns0nwjrf73wlf7m0000gp/T/ Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.QC4VUOGT8K/Render TERM_PROGRAM_VERSION=421.1 TERM_SESSION_ID=78E40B59-9BB1-49D9-B6E0-6F5CCBDC3C26 USER=jason SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.ZdYIe0WRYK/Listeners PATH=/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin PWD=/Users/jason/GIT/pcs-cli/publish LANG=en_US.UTF-8 XPC_FLAGS=0x0 XPC_SERVICENAME=0 SHLVL=1 HOME=/Users/jason LOGNAME=jason DISPLAY=/private/tmp/com.apple.launchd.IKSK4VyOZ5/org.macosforge.xquartz:0 =/usr/bin/ENV OLDPWD=/Users/jason/GIT/pcs-cli

[jasonb815]

Looks like it's not getting set.. on OSX you need to use the "export" command and not "set".. I'll set it and try again.. ??

[jasonb815]

failed again...

pcs -s basic --versionOverride 1.0.2 --dockerTagOverride 1.0.2 -r dotnet ? Enter a solution name: mq-pi-beta ? Select a subscription: MachineQ ? Select a location: East US ? Enter prefix for .azurewebsites.net: mq-pi-beta ? Enter a user name for the virtual machine: mqadmin ? Enter a password for the virtual machine: ? Confirm your password: ✓ Application registered: https://portal.azure.com/comcastcorp.onmicrosoft.com#blade/Microsoft_AAD_IAM/ApplicationBlade/objectId/91a80c30-65ca-4323-b5f3-57d493ac0420/appId/4e065398-2630-4c52-9116-24a68522d584 ✓ Created resource group: https://portal.azure.com/comcastcorp.onmicrosoft.com#resource/subscriptions/ca4aac28-a514-4213-9cc4-6b7618fb063e/resourceGroups/mq-pi-beta ✓ Provisioning State: Succeeded Resource Type: Microsoft.EventHub/namespaces/eventhubs/authorizationRules ✓ Provisioning State: Succeeded Resource Type: Microsoft.TimeSeriesInsights/environments/accessPolicies ✕ Deployment failed { "status": "Failed", "error": { "code": "ResourceDeploymentFailure", "message": "The resource operation completed with terminal provisioning state 'Failed'.", "details": [ { "code": "VMExtensionProvisioningError", "message": "VM has reported a failure when processing extension 'scriptextensions'. Error message: \"Enable failed: processing file downloads failed: failed to download file[1]: failed to download file: unexpected status code: got=404 expected=200\"." } ] } }

JasonB-MBP-2018:pcs-cli jason$ env TERM_PROGRAM=Apple_Terminal SHELL=/bin/bash TERM=xterm-256color TMPDIR=/var/folders/rf/jdxqm_kn7ns0nwjrf73wlf7m0000gp/T/ Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.QC4VUOGT8K/Render TERM_PROGRAM_VERSION=421.1 OLDPWD=/Users/jason/GIT/pcs-cli/publish TERM_SESSION_ID=78E40B59-9BB1-49D9-B6E0-6F5CCBDC3C26 USER=jason PCS_AUTH_REQUIRED=false SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.ZdYIe0WRYK/Listeners PATH=/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin PWD=/Users/jason/GIT/pcs-cli LANG=en_US.UTF-8 XPC_FLAGS=0x0 XPC_SERVICENAME=0 SHLVL=1 HOME=/Users/jason LOGNAME=jason DISPLAY=/private/tmp/com.apple.launchd.IKSK4VyOZ5/org.macosforge.xquartz:0 =/usr/bin/env JasonB-MBP-2018:pcs-cli jason$

[jillcary]

@jasonb815 thank you so much for the additional information. I have a few more questions/clarifications about your situation. Given that you are not doing a local deployment with pcs -s local I think the issue may be that you are trying to disable auth on your local machine but for the remote endpoint, but I'd like to confirm that this is your issue.

1) It would be preferable and most stable to use 2.1.2, you stated above that you would like to use 1.0.2 but only because 2.1.2 would not allow you to disable auth locally causing a 502 error. Is that correct?

2) If that is the case, when disabling auth, you mentioned that you are doing so locally, can you confirm if your docker containers are running locally for your service or if you are trying to use containers running in a VM on Azure?

3) When you point the webui at your backend service, what address are you pointing them to?

If you need auth disabled to point your local webui to a backend with auth disabled, There are instructions for setting up your local dev envornment here. You can also try the following: 1) Deploy the latest version (v2.1.2) of Remote Monitoring via the CLI or from https://www.azureiotsolutions.com/Accelerators 2) When the deployment has finished, navigate to the VM in your resource group under {resource group name} > vm-{name} 3) (optional) If you deployed through solutions.com, you will need to set a password on your VM. Go to "Reset password" and set a username and password on your VM 4) Navigate to the "Serial Console" to access the command line for the VM. Sign in with your credentials. 5) Start the services with auth disabled with the following command:

cd /app
sudo ./start.sh --unsafe

6) Verify that auth is disabled by making a request to the telemetry endpoint. You can do this by using an HTTP client like Postman to make the following GET request:

GET https://{name-of-resource-group}.azurewebsites.net/telemetry/v1/status

If auth is disabled you should get a 200 OK message, if auth is still enabled, you will get 502 Bad Gateway.

[jasonb815]

1 - Yes, we would prefer 2.1.2, but it isn't working 2 - Containers on an Azure VM 3 - this is how we've been doing it from the start..

[jillcary]

Ok thank you for clarifying. Please SSH or use serial console to execute the sudo ./start.sh --unsafe command on your 2.1.2 deployment and report back if things still aren't working, I'll continue to try to repro your issue in the meantime.

[jasonb815]

I've done that a MILLION times.. it doesn't report any errors

[jillcary]

I'm sorry, I think I misunderstood that you mentioned you were not on a VM but on your local machine.

[jillcary]

Can you confirm if I understand your situation correctly? 1) You have a 2.1.2 deployment that has been working fine in the past with an endpoint address that you were using successfully before. 2) Recently when you stopped and restarted the containers while SSH-ed into the VM, "unsafe" mode stopped working, reporting 502 errors for your frontend. 3) Because of this you wish to revert to version 1.0.2 but the deployment from the CLI is failing currently.

[jasonb815]

We have never gotten 2.1.2 to work in our config. We have used 1.0.1 and 1.0.2 up until the cli stopped being able to deploy 1.0.2 here recently.

Acceptable solutions for me :

1- 2.1.2 in UNSAFE mode - currently isn't working and we are getting a 502 in UNSAFE mode 2- 1.0.2 in UNSAFE mode - currently works, but we are unable to deploy any additional 1.0.2 PCS

[jillcary]

Thank you.

In order to help address #1, can you please log in to your VM and let us know what the output of the following command is? (From the VM running the 2.1.2 RM containers): docker ps -a

[jasonb815]

root@vm-txu6f:/app# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d0b948ccfe6e azureiotpcs/remote-monitoring-nginx:2.1.0 "/bin/bash /app/run.…" 27 hours ago Up 27 hours 80/tcp, 0.0.0.0:80->10080/tcp, 0.0.0.0:443->10443/tcp app_reverseproxy_1 3eded5f10ebe azureiotpcs/pcs-remote-monitoring-webui:2.1.0 "/bin/sh /app/run.sh" 27 hours ago Up 27 hours 10080/tcp, 10443/tcp app_webui_1 f1a2e3057055 azureiotpcs/asa-manager-dotnet:2.1.0 "/bin/bash /app/run.…" 27 hours ago Up 27 hours app_asamanager_1 d2f58f7f1c3d azureiotpcs/pcs-diagnostics-dotnet:2.1.0 "/bin/bash /app/run.…" 27 hours ago Up 27 hours app_diagnostics_1 67924377eef0 azureiotpcs/pcs-config-dotnet:2.1.0 "/bin/bash /app/run.…" 27 hours ago Up 27 hours app_config_1 6833635051e6 azureiotpcs/device-simulation-dotnet:DS-1.0.2 "/bin/bash /app/run.…" 27 hours ago Up 27 hours app_devicesimulation_1 4d195ec924c2 azureiotpcs/iothub-manager-dotnet:2.1.0 "/bin/bash /app/run.…" 27 hours ago Up 27 hours app_iothubmanager_1 032fa89fc63f azureiotpcs/telemetry-dotnet:2.1.0 "/bin/bash /app/run.…" 27 hours ago Up 27 hours app_telemetry_1 461c35f5d9d8 azureiotpcs/pcs-storage-adapter-dotnet:2.1.0 "/bin/bash /app/run.…" 27 hours ago Up 27 hours app_storageadapter_1 11b9d7f15e6e azureiotpcs/pcs-auth-dotnet:2.1.0 "/bin/bash /app/run.…" 27 hours ago Up 27 hours app_auth_1

[jillcary]

And when you type ls from the VM with the 2.1.2 containers in the /app path are you missing any of the following files?

certs
setup.log
webui-config.js
webui-config.js.safe
webui-config.js.unsafe
docker-compose.yml
simulate.sh
stop.sh
start.sh
update.sh
status.sh
logs.sh
stats.sh  
env-vars

[jasonb815]

from /app :

root@vm-txu6f:/app# ls -l total 120 drwxr-xr-x 2 root root 4096 Dec 12 21:16 certs -rw-r--r-- 1 root root 5336 Dec 12 21:16 docker-compose.yml -r--r----- 1 root root 4202 Dec 12 21:16 env-vars -rwxr-x--- 1 root root 112 Dec 12 21:16 logs.sh -rw-rw---- 1 root root 55353 Dec 12 21:16 setup.log -rwxr-x--- 1 root root 544 Dec 12 21:16 simulate.sh -rwxr-x--- 1 root root 1458 Dec 12 21:16 start.sh -rwxr-x--- 1 root root 41 Dec 12 21:16 stats.sh -rwxr-x--- 1 root root 201 Dec 12 21:16 status.sh -rwxr-x--- 1 root root 89 Dec 12 21:16 stop.sh -rwxr-x--- 1 root root 1219 Dec 12 21:16 update.sh -r--r--r-- 1 root root 238 Dec 12 21:16 webui-config.js -r--r--r-- 1 root root 238 Dec 12 21:16 webui-config.js.safe -r--r--r-- 1 root root 239 Dec 12 21:16 webui-config.js.unsafe

[jillcary]

Based on the output from the docker containers above, it looks like the version of the containers on your VM is 2.1.0. Are you able to deploy Remote Monitoring version 2.1.2 successfully via the pcs-cli?

[jasonb815]

I haven't tried via the CLI.. just the website..

[jasonb815]

what command would you like for me to try?

[jasonb815]

pcs -s basic -t remotemonitoring

[jasonb815]

it's still running but failing on lots of things... ??

[jasonb815]

✕ Deployment failed { "code": "Conflict", "message": "The Stream Analytics job is not in a valid state to perform this operation.", "details": { "code": "409", "message": "The Stream Analytics job is not in a valid state to perform this operation.", "correlationId": "f3300f55-13a6-4904-864b-d1b7551a4781", "requestId": "9c4a7f4d-7fdc-42c2-a185-3850138d7afb" } } { "error": { "code": "InvalidInput", "message": "The 'principalObjectId' access policy property is immutable.", "innerError": { "code": "ImmutableProperty" } } } { "status": "Failed", "error": { "code": "ResourceDeploymentFailure", "message": "The resource operation completed with terminal provisioning state 'Failed'." } } { "error": { "code": "PropertyChangeNotAllowed", "message": "Changing property 'adminUsername' is not allowed.", "target": "adminUsername" } }

[jillcary]

The website, as of today, is deploying RM version, 2.1.2. Please deploy a remote monitoring basic dotnet solution via www.azureiotsolutions.com, that way I can make sure that I can repro the exact behavior you are seeing as well.

[jasonb815]

update :

I updated my PCS CLI to include your most recent check-in from Friday at 1:21pm and now when I deploy a 2.x PCS it gets version 2.1.2rc1 and, and then I put it into UNSAFE mode, IT WORKS! Not sure what changed, but it is now working for me.. :)

thanks!

[jillcary]

@jasonb815 thanks so much for all of the detailed information. I'm very glad it works and please reopen or create a new issue if you see this behavior again.

[jasonb815]

Will we have the ability going forward to choose a specific version of PCS? We want to be able to "lock in" with a specific version for future deployments as we will need to spin up a new PCS for each new deployment of our solution. Does that make sense?

[sakimsft]

@jasonb815
PCS CLI is npm package and hence you can install to a specific version using below command: npm install -g \package\>@\<version\

where, package name is "iot-solutions" and version information you can find at here: https://www.npmjs.com/package/iot-solutions e.g., npm install -g iot-solutions@2.1.2

After above, you can verify: pcs --version

[jasonb815]

Sorry, I didn’t properly explain. I need to be able to install specific versions of the ENTIRE PCS, not the PCS CLI tool. ☺ We used to be able to do this with something like :

· pcs -s basic --versionOverride 1.0.2 --dockerTagOverride 1.0.2 -r dotnet

???