Iot edge not pulling image from Docker container registry

manueljesusrd commented 5 years ago

I've been all day dealing with a problem related to the deployment of custom modules in IoT Edge. I'm using a private Docker Container Registry (docker.io) as my container instead of Azure's, and I'm trying to deploy a custom module (the famous SampleModule) from Visual Studio Code. The problem is that once the deployment file has been pushed to Azure Portal, the custom module never gets deployed in my iot edge.

The process is:

Prepare the deployment template -> OK
Build and Push IoT Edge Solution - OK
Deployment for single device -> OK
Iot edge pulls image from Docker CR -> ERROR

I'm using Ubuntu 18.04, Visual Studio Code 1.38.1 (with all the latest Azure iot addons from the tutorials) and the latest versions of all Azure Iot components.

The output from the 'iot check' is:

Configuration checks
--------------------
√ config.yaml is well-formed - OK
√ config.yaml has well-formed connection string - OK
√ container engine is installed and functional - OK
√ config.yaml has correct hostname - OK
√ config.yaml has correct URIs for daemon mgmt endpoint - OK
√ latest security daemon - OK
√ host time is close to real time - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
√ production readiness: certificates - OK
√ production readiness: certificates expiry - OK
‼ production readiness: container engine - Warning
    Device is not using a production-supported container engine (moby-engine).
    Please see https://aka.ms/iotedge-prod-checklist-moby for details.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.

Connectivity checks
-------------------
√ host can connect to and perform TLS handshake with IoT Hub AMQP port - OK
√ host can connect to and perform TLS handshake with IoT Hub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with IoT Hub MQTT port - OK
√ container on the default network can connect to IoT Hub AMQP port - OK
√ container on the default network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the default network can connect to IoT Hub MQTT port - OK
√ container on the IoT Edge module network can connect to IoT Hub AMQP port - OK
√ container on the IoT Edge module network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to IoT Hub MQTT port - OK
√ Edge Hub can bind to ports on host - OK

20 check(s) succeeded.
3 check(s) raised warnings. Re-run with --verbose for more details.

I've made some research about what could be happening. I checked the container id from 'docker ps' and see the logPath to check it out while I deployed the module. This is what I could find:

{"log":"\u003c6\u003e 2019-09-19 17:35:55.580 +00:00 [INF] - Executing command: \"Command Group: (\\n  [Create module TestMessages]\\n  [Start module TestMessages]\\n)\"\n","stream":"stdout","time":"2019-09-19T17:35:55.580344648Z"}
{"log":"\u003c6\u003e 2019-09-19 17:35:55.580 +00:00 [INF] - Executing command: \"Create module TestMessages\"\n","stream":"stdout","time":"2019-09-19T17:35:55.580358567Z"}
{"log":"\u003c3\u003e 2019-09-19 17:35:58.574 +00:00 [ERR] - Executing command for operation [\"create\"] failed.\n","stream":"stdout","time":"2019-09-19T17:35:58.575074089Z"}
{"log":"Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling Create module TestMessages: Could not create module TestMessages\n","stream":"stdout","time":"2019-09-19T17:35:58.575096846Z"}
{"log":"\u0009caused by: Could not pull image manueljesusdelgado/modulereg:0.0.5-amd64\n","stream":"stdout","time":"2019-09-19T17:35:58.575101266Z"}
{"log":"\u0009caused by: pull access denied for manueljesusdelgado/modulereg, repository does not exist or may require 'docker login': denied: requested access to the resource is denied, StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2019_01_30.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2019_01_30/ModuleManagementHttpClient.cs:line 194\n","stream":"stdout","time":"2019-09-19T17:35:58.575104103Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 124\n","stream":"stdout","time":"2019-09-19T17:35:58.575126297Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2019_01_30.ModuleManagementHttpClient.CreateModuleAsync(ModuleSpec moduleSpec) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2019_01_30/ModuleManagementHttpClient.cs:line 96\n","stream":"stdout","time":"2019-09-19T17:35:58.575129245Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60\n","stream":"stdout","time":"2019-09-19T17:35:58.57513216Z"}
{"log":"\u003c3\u003e 2019-09-19 17:35:58.575 +00:00 [ERR] - Executing command for operation [\"Command Group: (\\n  [Create module TestMessages]\\n  [Start module TestMessages]\\n)\"] failed.\n","stream":"stdout","time":"2019-09-19T17:35:58.577598612Z"}
{"log":"Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling Create module TestMessages: Could not create module TestMessages\n","stream":"stdout","time":"2019-09-19T17:35:58.577621419Z"}
{"log":"\u0009caused by: Could not pull image manueljesusdelgado/modulereg:0.0.5-amd64\n","stream":"stdout","time":"2019-09-19T17:35:58.57762866Z"}
{"log":"\u0009caused by: pull access denied for manueljesusdelgado/modulereg, repository does not exist or may require 'docker login': denied: requested access to the resource is denied, StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2019_01_30.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2019_01_30/ModuleManagementHttpClient.cs:line 194\n","stream":"stdout","time":"2019-09-19T17:35:58.577632203Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 124\n","stream":"stdout","time":"2019-09-19T17:35:58.577711951Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2019_01_30.ModuleManagementHttpClient.CreateModuleAsync(ModuleSpec moduleSpec) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2019_01_30/ModuleManagementHttpClient.cs:line 96\n","stream":"stdout","time":"2019-09-19T17:35:58.577718101Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60\n","stream":"stdout","time":"2019-09-19T17:35:58.577720847Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35\n","stream":"stdout","time":"2019-09-19T17:35:58.577723398Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60\n","stream":"stdout","time":"2019-09-19T17:35:58.577727085Z"}
{"log":"\u003c3\u003e 2019-09-19 17:35:58.577 +00:00 [ERR] - Step failed in deployment 30, continuing execution. Failure when running command Command Group: (\n","stream":"stdout","time":"2019-09-19T17:35:58.57772976Z"}
{"log":"  [Create module TestMessages]\n","stream":"stdout","time":"2019-09-19T17:35:58.577732368Z"}
{"log":"  [Start module TestMessages]\n","stream":"stdout","time":"2019-09-19T17:35:58.57773465Z"}
{"log":"). Will retry in -01s.\n","stream":"stdout","time":"2019-09-19T17:35:58.577736855Z"}

As you can see, it seems to be a problem related to when iotedge should be login into Docker CR. I have username and password specified in the .env file in Visual Studio Code, and I'm perfectly able to push the image to the Docker CR from there. I'm also logged in from the Ubuntu terminal.

These are my json files related to the modules and the container registry:

modulse.json

{
  "$schema-version": "0.0.1",
  "description": "",
  "image": {
    "repository": "manueljesusdelgado/modulereg",
    "tag": {
      "version": "0.0.5",
      "platforms": {
        "amd64": "./Dockerfile.amd64",
        "amd64.debug": "./Dockerfile.amd64.debug",
        "arm32v7": "./Dockerfile.arm32v7",
        "arm32v7.debug": "./Dockerfile.arm32v7.debug"
      }
    },
    "buildOptions": []
  },
  "language": "java"
}

and the piece of deploymet.template.json

"settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {
              "manueljesusdelgado": {
                "username": "$CONTAINER_REGISTRY_USERNAME_manueljesusdelgado",
                "password": "$CONTAINER_REGISTRY_PASSWORD_manueljesusdelgado",
                "address": "docker.io"
              }
            }
          }

Device (Host) Operating System

Ubuntu 18.04

Architecture

amd64

Container Operating System

Linux containers

Runtime Versions

iotedged

iotedge 1.0.8 (208b2204fd30e856d00b280112422130c104b9f0)

Edge Agent

1.0.0

Edge Hub

1.0.0

Docker

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:29:11 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:27:45 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Additional Info

I've tried all the procedure with the Azure CR and it works fine. The problems seems to come only from when iotedge tries to login to Docker CR.

myagley commented 5 years ago

Can you try using https://index.docker.io/v1/ as the address for the Docker Hub registry instead of docker.io?

arsing commented 5 years ago

And double-check that "$CONTAINER_REGISTRY_USERNAME_manueljesusdelgado" and "$CONTAINER_REGISTRY_PASSWORD_manueljesusdelgado" were correctly populated in the actual deployment.

manueljesusrd commented 5 years ago

I tried what you two said, and we obtained no successful results :( Anyway, we switched to Azure CR and now everything is just fine 👍 Thanks!

V4A001 commented 5 years ago

And double-check that "$CONTAINER_REGISTRY_USERNAME_manueljesusdelgado" and "$CONTAINER_REGISTRY_PASSWORD_manueljesusdelgado" were correctly populated in the actual deployment.

Why do you use your name in the settings?

ianchanning commented 8 months ago

Try doing a docker login with the credentials you have and then a docker pull - this should show up any errors.

Azure / iotedge