search

Azure / iotedge

The IoT Edge OSS project
MIT License
1.46k stars 461 forks source link

Twin property not accepting longer than 512 characters #207

Closed MaddenJ603 closed 5 years ago

MaddenJ603 commented 6 years ago

Expected Behavior

According to this article: https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-module-twins All string values can be at most 4 KB in length.

Current Behavior

When my twin property exceeds 512 I receive this error message: [Edge] ErrorCode:ArgumentInvalid;Property or Tag value should be maximum 512 bytes.

Steps to Reproduce

  1. From VSCode edit deployment.json with a twin property for a module larger than 512 bytes
  2. Try to deploy to single device 3.Recieve above error message

Context (Environment)

Device (Host) Operating System

Windows 10 Pro

Architecture

amd64

Container Operating System

Linux

Runtime Versions

iotedged

iotedge 1.0.0 (52ef77db24126bf473265fc09c53d35290a2dd6b)

Edge Agent

mcr.microsoft.com/azureiotedge-agent:1.0.1

Edge Hub

mcr.microsoft.com/azureiotedge-hub:1.0.1

Docker

Client: Version: 18.06.0-ce API version: 1.38 Go version: go1.10.3 Git commit: 0ffa825 Built: Wed Jul 18 19:05:28 2018 OS/Arch: windows/amd64 Experimental: false

Server: Engine: Version: 18.06.0-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: 0ffa825 Built: Wed Jul 18 19:13:46 2018 OS/Arch: linux/amd64 Experimental: true

Logs

[Edge] Start deployment to device [JMEdge3] [Edge] Deployment failed. Error: Request failed with status code 400 [Edge] ErrorCode:ArgumentInvalid;Property or Tag value should be maximum 512 bytes. Error in.....

Additional Information

myagley commented 6 years ago

Hi. Thanks for the issue report.

Where is the error message being presented? Is this coming from the VSCode console? We recently expanded this limit in the Edge Hub itself for version 1.0.1 (https://github.com/Azure/iotedge/commit/2590d7e87db3dd0fbd21e15cfa441dfde22f4a52).

MaddenJ603 commented 6 years ago

Yes in VS Code console.

If I ‘edit module twin’ within VSCode and add a long value for the property I get a rather simple error return when I try to save: [ModuleTwin] Failed to update Module Twin: Error: Request failed with status code 400

If I try to ‘Create Deployment for Single Device’ referncing a deploy.config that has a long string for a twin property, I get the error messages I put in the original post.

Of note, I’m using a ‘special’ version of the IOTHub extension that Jon Gallant sent me to be able to compile shared libraries. Is it possible it is caused by the extension?

John Madden 281-620-7284

From: Mike Yagley Sent: Friday, August 24, 2018 10:27 To: Azure/iotedge Cc: MaddenJ603; Author Subject: Re: [Azure/iotedge] Twin property not accepting longer than 512characters (#207)

Hi. Thanks for the issue report. Where is the error message being presented? Is this coming from the VSCode console? We recently expanded this limit in the Edge Hub itself for version 1.0.1 (2590d7e). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

myagley commented 6 years ago

Apparently, this change wasn't rolled out to all IoT Hub due to technical reasons. However, the documentation was updated to increase the limit from 512b to 4kb. This is a miss on our part and I apologize.

There is desire to expand this limit, but at this point, we need to assume that the limit is 512b, until the IoT Hub service can be updated. We are currently updating the documentation to match the reality in the IoT Hub service. I apologize again for the confusion here.

MaddenJ603 commented 6 years ago

Any word on how long that rollout will take? eg: Days, or weeks?

Jazmine9 commented 6 years ago

I get the same error when I set a long createOptions string while deploying the module.

Failed To Set Modules ‎ BadRequest:{ "Message": "ErrorCode:ArgumentInvalid;Property or Tag value should be maximum 512 bytes. Error in Property/Tag

Jan-Olof commented 6 years ago

I think the same problem is happening here:

https://github.com/Azure/iot-edge-opc-publisher/issues/143

BewaAutomatisierung-RD commented 5 years ago

@myagley - Any news on when the limit for create options will be increased ? Using the OpcPublisher module there are many options to be set and the 512 bytes are not really sufficient!

myagley commented 5 years ago

We added support for chunking the createOptions in October. The spec for this chunking is here: https://github.com/Azure/iotedge/pull/396

The portal manages this chunking automatically. What tooling do you typically use to deploy? (Portal, VSCode, azure cli)

tristanbarcelon commented 5 years ago

@myagley I am seeing this error too when deploying a release through Azure Devops release management. Deploy is running on Hosted Ubuntu 1604 with the following error in log. Create property values were sanitized but properties were kept. Edge agent version in /etc/iotedge/config.yaml is 1.0.5 and deployment json is referencing the image mcr.microsoft.com/azureiotedge-agent:1.0.5 so they are identical.

2019-07-22T13:09:22.1545088Z ==============================================================================
2019-07-22T13:09:22.1545203Z Task         : Azure IoT Edge
2019-07-22T13:09:22.1545332Z Description  : Build and deploy an Azure IoT Edge image
2019-07-22T13:09:22.1545407Z Version      : 2.2.0
2019-07-22T13:09:22.1545562Z Author       : Microsoft Corporation
2019-07-22T13:09:22.1545649Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
2019-07-22T13:09:22.1545800Z ==============================================================================
2019-07-22T13:09:22.4624159Z Start deploying...
2019-07-22T13:09:22.4634342Z Deployment task is running in build pipeline? false
2019-07-22T13:09:22.4640674Z /home/vsts/work/r1/a/_my-artifact/drop/deployment.amd64.json
2019-07-22T13:09:22.4641615Z Checking if the following file is a valid json: /home/vsts/work/r1/a/_my-artifact/drop/deployment.amd64.json
2019-07-22T13:09:22.4651471Z Valid
2019-07-22T13:09:27.8358808Z Normalized deployment id is: my-artifact-release-3-12942
2019-07-22T13:09:49.0517850Z [command]/usr/bin/az iot edge deployment create --config-id my-artifact-release-3-12942 --hub-name my-iot-hub-01 --content /tmp/deployment_1563800967834.json --target-condition deviceId='testiotedgedevice' --priority 0
2019-07-22T13:09:50.8005325Z ERROR: {'Message': 'ErrorCode:ArgumentInvalid;Property or Tag value should be maximum 512 bytes. Error in Property/Tag {"Env":["ApplicationInsightsKey=guid","HealthCheckInterval=60","LogEventLevel=Debug","SqlConnection=Data Source=mysqlserver.corp.mydomain.org;Initial Catalog=mydatabase;User ID=testuser;Password=testpassword","DomainName=mydevice.domain","BrokerName=testdevicebrokername","SendToRabbitMQ=false","RabbitMq_QueueName=MessageBroker.Endpoint","RabbitMQServer=myrabbitmq.corp.mydomain.org","RabbitMQServerPort=5672","RabbitMQUserName=testrabbitmquser","RabbitMQPassword=***","Rabbi', 'ExceptionMessage': 'Tracking ID:a10a88aacfb24e2a98db1625ff8d63d3-G:13-TimeStamp:07/22/2019 13:09:50'}
2019-07-22T13:09:51.7674450Z ##[error]Error: Error: The process '/usr/bin/az' failed with exit code 1
2019-07-22T13:10:51.9822210Z ##[section]Finishing: Azure IoT Edge - Deploy to IoT Edge devices
veyalla commented 5 years ago

@shizn - is this something you can help with?

shizn commented 5 years ago

The Azure DevOps tasks is based on az-cli, we will try to reproduce this and get back to you soon.

adashen commented 5 years ago

@tristanbarcelon Can you share your deployment template? It will help us to investigate the issue.

tristanbarcelon commented 5 years ago

@adashen , let me see how much of my deployment.template.json I can post without disclosing product-specific information. Alternatively, are you able to send a teams meeting request to tristan_barcelon@jabil.com? @blackchoey used teams before to diagnose a bug with iotedgedev cli.

tristanbarcelon commented 5 years ago

@adashen, here's the deployment template we are using. We use Azure Devops for build and deployment. Variables in the template (starting with $) are replaced with #{ VARIABLE_NAME} during build and published as an artifact. At release creation, these variables will be replaced using token replacement task with values defined in variable groups scoped to a specific stage. It is during deployment of that release that we encounter the error.

{
  "$schema-template": "1.0.0",
  "modulesContent": {
    "$edgeAgent": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "runtime": {
          "type": "docker",
          "settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {
              "testregistry": {
                "username": "myregistry",
                "password": "myregistrypassword",
                "address": "myregistry.azurecr.io"
              }
            }
          }
        },
        "systemModules": {
          "edgeAgent": {
            "type": "docker",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-agent:1.0.5",
              "createOptions": {}
            }
          },
          "edgeHub": {
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-hub:1.0.5",
              "createOptions": {
                "HostConfig": {
                  "PortBindings": {
                    "5671/tcp": [
                      {
                        "HostPort": "5671"
                      }
                    ],
                    "8883/tcp": [
                      {
                        "HostPort": "8883"
                      }
                    ],
                    "443/tcp": [
                      {
                        "HostPort": "443"
                      }
                    ]
                  }
                }
              }
            }
          }
        },
        "modules": {
          "TEST1_MCC": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "${MODULES.TEST1_MCC}",
              "createOptions": {
                "Env": [
                  "ApplicationInsightsKey=$CC_IFACTORY_APP_INSIGHTS_KEY",
                  "HealthCheckInterval=$CC_HEALTH_CHECK_INTERVAL",
                  "LogEventLevel=$CC_LOG_LEVEL",
                  "CamxSqlConnection=$CAMX_SQL_CONNECTION",
                  "DomainName=$DOMAIN_NAME",
                  "BrokerName=$BROKER_NAME",
                  "SendToRabbitMQ=$SEND_TO_RABBIT_MQ",
                  "RabbitMq_QueueName=$RABBITMQ_QUEUE_NAME",
                  "RabbitMQServer=$RABBITMQ_SERVER",
                  "RabbitMQServerPort=$RABBITMQ_SERVER_HOST",
                  "RabbitMQUserName=$RABBITMQ_USERNAME",
                  "RabbitMQPassword=$RABBITMQ_PASSWORD",
                  "RabbitMQVirtualHost=$RABBITMQ_VIRTUALHOST"
                ],
                "HostConfig": {
                  "PortBindings": {
                    "80/tcp": [
                      {
                        "HostPort": "8010"
                      }
                    ]
                  }
                }
              }
            }
          },
          "TEST2_TXC": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "${MODULES.TEST2_TXC}",
              "createOptions": {
                "Env": [
                  "ApplicationInsightsKey=$CC_IFACTORY_APP_INSIGHTS_KEY",
                  "HealthCheckInterval=$CC_HEALTH_CHECK_INTERVAL",
                  "LogEventLevel=$CC_LOG_LEVEL",
                  "CamxSqlConnection=$CAMX_SQL_CONNECTION",
                  "DomainName=$DOMAIN_NAME",
                  "BrokerName=$BROKER_NAME",
                  "EqMonClient=$EQUIP_MONITOR_CLIENT"
                ]
              }
            }
          }
        }
      }
    },
    "$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "routes": {
          "CAMX_MCCToCAMX_TXC": "FROM /messages/modules/CAMX_MCC/outputs/CamxRawMsg INTO BrokeredEndpoint(\"/modules/CAMX_TXC/inputs/CamxRawMsg\")",          
          "CAMX_MCCToIoTHub": "FROM /messages/modules/CAMX_TXC/outputs/* INTO $upstream"
        },
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200
        }
      }
    }
  }
}
adashen commented 5 years ago

@tristanbarcelon Thanks for the information. We will test your manifest and keep you updated.

tristanbarcelon commented 5 years ago

@adashen I saw discussion on this pull request which could solve the issue at deployment time. Since the PR has been merged, how can we know the version of tooling that contains this change? As far as I know, iotedgedev cli is what's used on a developer's environment to build the deployment template. Is it the same for build servers when we use build and deploy commands in the Azure IOT Edge task? iotedgedev 2.0.1 pip module is used during build and I was expecting the chunked createOptions to appear in deployment.json.

adashen commented 5 years ago

@tristanbarcelon, Yes Azure OT Edge task uses iotedgedev cli in the back end. And it should chunk the createOptions to 512 size limit. And I tested your deployment template with iotedgedev cli. And saw the createOptions has been chunked and could deploy successfully. So the root cause may in the Azure edge pipeline task. We will investigate further and keep you updated.

tristanbarcelon commented 5 years ago

@adashen Do you have workarounds we can use at the moment to get past this inability to deploy?

adashen commented 5 years ago

@tristanbarcelon From the last comment you are using token replacement during release right? That may be the root cause, when generating the deployment manifest, the build/push task handle the chunk for you and make sure the deployment manifest createOptions are chunked. However, you are using token replacement during release, means the size of createOptions are changed after chunking, and it exceeds 512.

tristanbarcelon commented 5 years ago

@adashen that is correct. We are using token replacement task during release/deploy and not during build/push. We do this because we need to build edge modules and publish them without hardcoding deployment.json files to specific values during build/publish. Delaying this replacement as late as possible allows us to deploy modules to any environment we wish simply by changing createOptions or desiredProperties values. I think it would be very useful to specify a "chunk" flag during build so it would chunk createOption into a separate line per property and value pair. Even better would be to make this the default behavior due to the 512 char limit on twin properties.

tristanbarcelon commented 5 years ago

@adashen We also have a unique scenario that requires us to build all modules that are in a solution BUT dynamically compose at deployment time the modules that will be deployed. Each of our modules tend to focus on a particular type of equipment (spanning several compatible models) from a given vendor. Given that our target environments can have different combinations of equipment from multiple vendors, we need to be able to recompose the original deployment template json modules at deployment time to only include modules we need. Are there built-in functions for this or do we need to write our own?

blackchoey commented 5 years ago

@tristanbarcelon the iotedgedev will chunk the createOptions when generating deployment manifest. Replacing tokens in the generated deployment manifest may cause the size of createOptions exceeds 512 characters. That's why you're seeing such errors. Could you try to use Generate Deployment Manifest in the Azure IoT Edge task to generate deployment manifest during release? This task can replace environment variables in deployment template (in the form of $ENV_NAME or ${ENV_NAME}) so you can use it to generate deployment manifest dynamically before deploy. This also allows you have multiple deployment templates and choose which template to deploy in the release pipeline. Please kindly notice that the Generate Deployment Manifest task also needs the module.json files in order to expand the image placeholders (e.g. ${MODULES.TEST2_TXC}). Please make sure the module.json exists in the modules folder.

tristanbarcelon commented 5 years ago

@blackchoey To perform what you are suggesting, what are the minimum set of files I'd need to publish in the artifact when using Generate Deployment Manifest in a release definition? Our build definition is more or less composed of the following steps:

Our release definition triggers off of the deployment manifest published as artifact from build.

I just looked at the parameters for Generate deployment manifest action and it is very similar to Build module images with the only additional parameter being the output path for deployment.json. Doesn't the Build module images action perform the equivalent action as Generate deployment manifest in addition to building docker containers out of each subfolder in modules that is in the deployment template? Are you asking me to use Generate deployment manifest action on the published artifact ( tokenized deployment manifest file ) after I fix the deployment variabls so they are in the form $ENV_NAME or ${ENV_NAME} ? Seems like I would need to publish entire source (excluding .cs and .csproj files) from repo as artifact in order for the task to expand the image placeholders properly.

blackchoey commented 5 years ago

@tristanbarcelon You just need the module.json and deployment.template.json in order to generate deployment manifest. Here's a sample yaml snippet to copy related files:

steps:
- task: CopyFiles@2
  displayName: 'Copy Files to: Drop folder'
  inputs:
    Contents: |
     deployment.template.json
     **/module.json
    TargetFolder: '$(Build.ArtifactStagingDirectory)'

The reason we introduce the Generate deployment manifest task is that putting deployment manifest credentials in drop folder is not a good practice. We hope the new task can give you more flexibility to generate deployment manifest. For example, you can fill a registry credential with only pull permission during release, which provided enhanced security for the deployment.

tristanbarcelon commented 5 years ago

@blackchoey Thank you for your feedback. Is the default relative path between template and module.json files always similar to how it looks below? Are there built-in Azure Devops tasks that I can use during build to help me package just these select files with the structure intact?

deployment.template.json
modules
   |--- MODULE1
   |       |-- module.json
   |--- MODULE2
           |-- module.json
blackchoey commented 5 years ago

@tristanbarcelon If you're using VS Code IoT Edge extension to create and develop IoT Edge solution, the default relative path between template and module.json is what you see now. The ${MODULES.MODULE1} placeholder in deployment template means the module located at modules/MODULE1 folder. We don't have a task to copy these files for now.

lt72 commented 5 years ago

Closing issue as it seems to be stale, please re-open if necessary.

UAQureshi commented 2 years ago

``I am facing exactly similar issue after replacing provisioning token for an Azure IoT edge module in deployment template. Steps I am doing are as follows:

error looks like `this:

2021-12-11T22:21:00.1939623Z [command]/usr/bin/az iot edge deployment create --deployment-id xxxxxxxx-devops-deployment --hub-name xxxxxx-dev-iothub --content /tmp/deployment_1639261241257.json --target-condition deviceId='dev-iotedge' --priority 0 --output none 2021-12-11T22:21:02.3577847Z ERROR: {'Message': 'ErrorCode:ArgumentInvalid;Property or Tag name should be maximum 1024 bytes. Error in Property/Tag eyJhbGciOiJSUzI1NiIsImtpZCI6InNmVkRoUzYtNkNLcjZnUkg2cDd0RWhxRUpNZyIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJBVkFlZGdlIiwiYXVkIjpbImh0dHBzOi8vYzIxYWMyOGU2OGQ0NGJiOGExYWNhZGE5YzgzMzNjNzAuZWRnZWFwaS53ZXN0ZXVyb3BlLnZpZGVvYW5hbHl6ZXIuYXp1cmUubmV0L2VkZ2VNb2R1bGVzL0FWQWVkZ2UiXSwiZXhwIjoxNjM5ODY1OTY5LCJ0b2tlblR5cGUiOiJQcm92aXNpb25pbmdUb2tlbiIsIm1vZHVsZUFybUlkIjoiL3N1YnNjcmlwdGlvbnMvODdmMGI0MGYtYzY5OC00YmY1LWIxMjItYWZmMjcwZjY0NmU3L3Jlc291cmNlR3JvdXBzL2F6dXJlc3RhY2tlZGdlLXBvYy1yZy9wcm92aWRlcnMvTWljcm9zb2Z0Lk1lZGlhL3ZpZGVvQW5hbHl6ZXJzL2RldmF2YWd0djVuYm03aWV4dDQvZWRnZU1vZHVsZXMvQVZBZWRnZSIsIm1vZHVsZUlkIjoiMmZhYmQ0NzQtYmM1Zi00NGI5LWEwMjQtMzEzZWQ0Y2ZlZWM3IiwiaXNzIjoiaHR0cHM6Ly93ZXN0ZXVyb3BlLnZpZGVvYW5hbHl6ZXIuYXp1cmUubmV0LyJ9.xOk_snVjjYogjovSHwuVTntTuSJ7siX54iB6wIa9bnBxd0wV9I9SQ_rj6HJdDMauDG-QJj4va_evSc6iWMY6Ak0z7iACkzCAbSXz0q8feTStomGO3PumE4_tlxU7rqei4G-EzU-WEBJt76rAUhiW8fIqQLnNdHUyGuXlWzCJk_IZCjf78EZF2oyv4puaPC047apOkzXllbDnNIe6t7oJdh8ARGcalz1X3zrQrp3yHC3ZpqPJXes4DO2BBXN9KdTqTMo1KGY8CemxDngldkRb0F8FaZLRsLmjfVq0YYhaGaLr9c4SLPyQCOVfJRHUxuevbEUQqSErxbY6GGsbHtB15A', 'ExceptionMessage': 'Tracking ID:30c5af487a3d4d1184bfacffd8541a23-G:0-TimeStamp:12/11/2021 22:21:02'} 2021-12-11T22:21:02.7944199Z ##[error]Error: Error: The process '/usr/bin/az' failed with exit code 1 2021-12-11T22:22:03.6175473Z ##[section]Finishing: Deploy to IoT Edge device`