Closed SandroVaroli closed 1 year ago
@onalante-msft is this fixed by your PR https://github.com/Azure/iot-identity-service/pull/451?
@SandroVaroli how urgent is this for you?
@jlian It's quite urgent since the script I shared is taken (a subset of...) from the one stored in the Linux golden image we used to install 40 devices that now are around the world ready to be installed in industrial plants. (a sort of "near to" plug and play provisioning that is possible just with TPM provisioning that does not need to share key) I cannot recall all of them in order to change the script forcing version 1.3...
I made tests on a spare device I have in my office... We could wait one week or so asking people to delay the switching on of new devices, but I have couples of urgent installation for some customer that cannot be delayed more than this.
Ok, I'll provide an update this afternoon. In the meantime
main
CI/CD here resolves the issue for you? As of today I think the latest one that works would be packages_ubuntu-20.04_amd64. Download and extract then sudo apt install aziot-identity-service_1.4.0~dev-1_amd64.deb
To add: it may also be worth trying device:/dev/tpmrm0
since the libtss version provided on Ubuntu 20.04 defaults to /dev/tpm0
, which has more usage caveats.
@onalante-msft so you think it's a different issue?
It may be, but it is still worth using the updated iot-identity-service
packages to remove the other TPM issue as a possible point of failure.
@jlian let me summarize what your suggestion is:
aziot-identity-service_1.4.0~dev-1_amd64.deb
packageThere is some step that is not fully convincing me...
How long we should wait for a working 1.4 version?
Some consideration
If the 1.4 has blocking bug... shouldn't be released or downloaded as last stable... the sudo apt-get install aziot-edge
should install version 1.3 until version 1.4 will be stable
Bugs happen. I make sw, so I know.... but also hotfixes exist, and can be applied to releases for blocking bugs...
You are out with a iotedge version that does not support TPM... add this should at least written on documentation...
@SandroVaroli sorry, I wasn't suggesting you distribute the dev build to your customers. The problem you're seeing isn't exactly the same as the one that @onalante-msft had recently fixed. So I was asking if you'd be able to help check that the latest .deb fixes your problem while we try dig in and try to repro your issue. Does that make more sense?
In the meantime you have a point re:
If the 1.4 has blocking bug... shouldn't be released or downloaded as last stable... the
sudo apt-get install aziot-edge
should install version 1.3 until version 1.4 will be stable Bugs happen. I make sw, so I know.... but also hotfixes exist, and can be applied to releases for blocking bugs... You are out with a iotedge version that does not support TPM... add this should at least written on documentation...
Adding @micahl and @damonbarry. Is this something we could do?
@jlian. Ok I'll try to install the update version in order to check if this fix the issue.
I realized that I never provided an update that I promised. The current plan is we will release a 1.4.1 as soon as we can, hopefully by middle of next week to align with the new .NET security patch that we also have to take.
Currently 1.4.1 has only one thing in scope and that is @onalante-msft's fix https://github.com/Azure/iot-identity-service/pull/451.
However like I said it's similar but not exactly the same issue you see. So:
I've edited the release notes to call out the fact that we're investigating issues related to TPM provisioning.
@jlian @onalante-msft I made several tests. the working one is:
sudo apt-get install aziot-edge
sudo dpkg -i ./aziot-identity-service_1.4.0-dev-1_amd64.deb
sudo apt-get install -f
because it will override the identity service with version 1.4.0It worked. That means:
So we can say the the issue is fixed with the update. If the version 1.4.1 will be released inside next week I can manage to delay the setup of the two devices of the plants currently in commissioning phase
Thanks for the support
Really appreciate your help and thanks for your patience!
CC: @damonbarry re: 1.4.1 TPM fix
1.4.1 is out https://github.com/Azure/azure-iotedge/releases/tag/1.4.1 @SandroVaroli
I'm closing the issue. If you see any problem we can reopen
I think I am seeing this issue still with this. aziot-tpmd[22949]: WARNING:tcti:src/tss2-tcti/tcti-device.c:428:Tss2_Tcti_Device_Init() Failed to open default TCTI device file /dev/tpmrm0: Permission denied
@tervoju at first glance that looks like a potentially different thing. Please open a separate issue with details on your setup and steps taken to reproduce the problem to assist us in narrowing down the issue.
One quick thing to check is that you’ve given IoT Edge access to the TPM by setting the tpmaccess.rules similar to what we have in docs.
ok, it looks indeed different thing now. got it little bit further but still fully working. will create another error if needed.
Expected Behavior
With version 1.3 we are able to deploy iotedge on a clean installation of ubuntu device with the next script using TPM provisioning:
and this is the result in the /etc/aziot/config.toml file
And this the result of iotedge system status
Of course on the DPS an individual enrollment with TPM method and the right Endorsement key has been prepared in advance.
Current Behavior
With version 1.4 the config.toml file slightly changed the format adding 2 rows in the TPM provisioning section where it is possible to send a custom payload during DPS registration... We have no need to do this (I suppose, since no change in DPS) so let them commented out. The script is identical with few changes on how config.toml is craeted (temporary solution... in the future will manage better) and of course no version forced. Here the differences
and this is the result in the /etc/aziot/config.toml file
Ok, this looks like perfectly identical to the one of version 1.3... greats.. it shuold work... Also everything is following documentation But if running on Ubuntu clean installation, it doesn't works
Here the result of iotedge systen status
Steps to Reproduce
Context (Environment)
Output of
iotedge check
Click here
``` iotedge check Configuration checks (aziot-identity-service) --------------------------------------------- √ keyd configuration is well-formed - OK √ certd configuration is well-formed - OK √ tpmd configuration is well-formed - OK √ identityd configuration is well-formed - OK √ daemon configurations up-to-date with config.toml - OK √ identityd config toml file specifies a valid hostname - OK √ aziot-identity-service package is up-to-date - OK √ host time is close to reference time - OK √ preloaded certificates are valid - OK √ keyd is running - OK √ certd is running - OK √ tpmd is running - OK √ identityd is running - OK × read all preloaded certificates from the Certificates Service - Error could not load cert with ID "aziot-edged-trust-bundle" Caused by: parameter "id" has an invalid value caused by: not found √ read all preloaded key pairs from the Keys Service - OK √ check all EST server URLs utilize HTTPS - OK √ ensure all preloaded certificates match preloaded private keys with the same ID - OK Connectivity checks (aziot-identity-service) -------------------------------------------- ‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning Could not retrieve iothub_hostname from provisioning file. Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information. Since no hostname is provided, all hub connectivity tests will be skipped. ‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning Could not retrieve iothub_hostname from provisioning file. Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information. Since no hostname is provided, all hub connectivity tests will be skipped. ‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning Could not retrieve iothub_hostname from provisioning file. Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information. Since no hostname is provided, all hub connectivity tests will be skipped. √ host can connect to and perform TLS handshake with DPS endpoint - OK Configuration checks -------------------- √ aziot-edged configuration is well-formed - OK √ configuration up-to-date with config.toml - OK √ container engine is installed and functional - OK × configuration has correct URIs for daemon mgmt endpoint - Error SocketError - SocketErrorCode (TimedOut) : Operation timed out One or more errors occurred. (Got bad response: ) √ aziot-edge package is up-to-date - OK √ container time is close to host time - OK ‼ DNS server - Warning Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub. Please see https://aka.ms/iotedge-prod-checklist-dns for best practices. You can ignore this warning if you are setting DNS server per module in the Edge deployment. ‼ production readiness: logs policy - Warning Container engine is not configured to rotate module logs which may cause it run out of disk space. Please see https://aka.ms/iotedge-prod-checklist-logs for best practices. You can ignore this warning if you are setting log policy per module in the Edge deployment. × production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error Could not check current state of edgeAgent container × production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error Could not check current state of edgeHub container √ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK Connectivity checks ------------------- 23 check(s) succeeded. 5 check(s) raised warnings. Re-run with --verbose for more details. 4 check(s) raised errors. Re-run with --verbose for more details. 7 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details. ```Device Information
Runtime Versions
Server: Engine: Version: 20.10.17+azure-3 API version: 1.41 (minimum version 1.12) Go version: go1.17.11 Git commit: a89b84221c8560e7a3dee2a653353429e7628424 Built: Mon Jun 6 22:32:38 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.5.13+azure-1 GitCommit: a17ec496a95e55601607ca50828147e8ccaeebf1 runc: Version: 1.0.3 GitCommit: f46b6ba2c9314cfc8caae24a32ec5fe9ef1059fe docker-init: Version: 0.19.0 GitCommit:
Sep 05 15:45:52 edgelinux aziot-identityd[4550]: 2022-09-05T15:45:52Z [INFO] - Starting service... Sep 05 15:45:52 edgelinux aziot-identityd[4550]: 2022-09-05T15:45:52Z [INFO] - Version - 1.4.0 Sep 05 15:45:52 edgelinux aziot-identityd[4550]: 2022-09-05T15:45:52Z [INFO] - Provisioning starting. Reason: Startup Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [INFO] - Starting service... Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [INFO] - Version - 1.4.0 Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: ERROR:tcti:src/tss2-tcti/tcti-device.c:439:Tss2_Tcti_Device_Init() Failed to open device file /dev/tpm0: Permission denied Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: WARNING:tcti:src/tss2-tcti/tctildr.c:62:tcti_from_init() TCTI init for function 0x7fdeeded3fb0 failed with a000a Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: WARNING:tcti:src/tss2-tcti/tctildr.c:92:tcti_from_info() Could not initialize TCTI named: tcti-device Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:150:tcti_from_file() Could not initialize TCTI file: device Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: ERROR:tcti:src/tss2-tcti/tctildr.c:418:Tss2_TctiLdr_Initialize_Ex() Failed to instantiate TCTI Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [ERR!] - service encountered an error Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [ERR!] - caused by: internal error Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [ERR!] - caused by: could not initialize TPM Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [ERR!] - caused by: tcti:IO failure Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [ERR!] - 0:
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 1:
Sep 05 15:45:52 edgelinux systemd[1]: aziot-tpmd.service: Main process exited, code=exited, status=1/FAILURE
Sep 05 15:45:52 edgelinux systemd[1]: aziot-tpmd.service: Failed with result 'exit-code'.
sudo apt-get autoremove --purge aziot-edge
get all container names, stop them all and delete them all
sudo docker stop $(sudo docker ps -a -q) sudo docker rm $(sudo docker ps -a -q)
get all docker images ids and delete them all
sudo docker rmi $(sudo docker images -a -q)
delete volumes not connected to containers
sudo docker volume ls -f dangling=true sudo docker volume prune
purge docker
sudo apt-get autoremove --purge moby-engine