search

eclipse-ditto / ditto

Eclipse Ditto™: Digital Twin framework of Eclipse IoT - main repository
https://eclipse.dev/ditto/
Eclipse Public License 2.0
689 stars 224 forks source link

Feature messages are not sent to the IoT device #1710

Closed dstryzhov closed 1 year ago

dstryzhov commented 1 year ago

In our production server, we have two IoT devices with same firmware version (therefore same Ditto connectivity code).

However, when trying to send a feature message, one of them does not receive anything.

Looking at the logs for the whole Ditto cluster, there is an obvious difference - looks like in case of failing device there is no HiveMqtt3PublisherActor log statement forwarding the event to MQTT.

Successful feature message flow:

Aug 2, 2023 @ 17:10:18.612","2023-08-02 16:10:18,612 INFO  [9fb1b00b-e0f6-4d7e-8407-d33f701db4f9] o.e.d.c.s.m.m.h.HiveMqtt3PublisherActor akka://ditto-cluster/system/sharding/connection/17/mqtt-connection/pa/$a/c1/HiveMqtt3PublisherActor1 - Publishing mapped message of type <messages.commands:featureMessage> to PublishTarget <MqttPublishTarget [topic=devices/com.companyname/firstThingId, qos=0]>
Aug 2, 2023 @ 17:10:18.596","2023-08-02 16:10:18,596 INFO  [9fb1b00b-e0f6-4d7e-8407-d33f701db4f9] o.e.d.c.a.a.ConciergeForwarderActor akka://ditto-cluster/user/gatewayRoot/conciergeForwarder - Forwarding signal with ID <com.companyname:firstThingId> and type <messages.commands:featureMessage> to concierge enforcer
Aug 2, 2023 @ 17:10:18.581","2023-08-02 16:10:18,581 DEBUG [] a.a.ActorSystemImpl akka.actor.ActorSystemImpl(ditto-cluster) - http-request: HttpRequest(HttpMethod(POST),http://ingress.companyname.com:8080/api/2/things/com.companyname:firstThingId/features/featureName/inbox/messages/initiateFeatureFlow?timeout=20,List(Timeout-Access: <function1>, Raw-Request-URI: /api/2/things/com.companyname:firstThingId/features/featureName/inbox/messages/initiateFeatureFlow?timeout=20, X-Forwared-User: ditto, x-ditto-pre-authenticated: xxx, Host, X-Real-Ip, X-Forwarded-For, Authorization, User-Agent: PostmanRuntime/7.32.3, Accept: */*, Accept-Encoding: gzip, deflate, br),HttpEntity.Strict(application/json,25 bytes total),HttpProtocol(HTTP/1.1))
Aug 2, 2023 @ 17:10:18.578","2023-08-02 16:10:18,577 DEBUG [] a.a.ActorSystemImpl akka.actor.ActorSystemImpl(ditto-cluster) - http: HttpRequest(HttpMethod(POST),http://ingress.companyname.com:8080/api/2/things/com.companyname:firstThingId/features/featureName/inbox/messages/initiateFeatureFlow?timeout=20,List(Timeout-Access: <function1>, Raw-Request-URI: /api/2/things/com.companyname:firstThingId/features/featureName/inbox/messages/initiateFeatureFlow?timeout=20, X-Forwared-User: ditto, x-ditto-pre-authenticated: xxx, Host, X-Real-Ip, X-Forwarded-For, Authorization, User-Agent: PostmanRuntime/7.32.3, Accept: */*, Accept-Encoding: gzip, deflate, br),HttpEntity.Strict(application/json,25 bytes total),HttpProtocol(HTTP/1.1))

Failed (undelivered) feature message:

Aug 2, 2023 @ 17:07:47.179","2023-08-02 16:07:47,179 INFO  [cd82c168-914a-4c0d-bf91-4918ae599e10] o.e.d.g.s.e.d.RequestResultLoggingDirective  - StatusCode of request POST '/api/2/things/com.companyname:secondThingId/features/featureName/inbox/messages/initiateFeatureFlow?timeout=20' was: 408
Aug 2, 2023 @ 17:07:27.162","2023-08-02 16:07:27,160 INFO  [cd82c168-914a-4c0d-bf91-4918ae599e10] o.e.d.c.a.a.ConciergeForwarderActor akka://ditto-cluster/user/gatewayRoot/conciergeForwarder - Forwarding signal with ID <com.companyname:secondThingId> and type <messages.commands:featureMessage> to concierge enforcer
Aug 2, 2023 @ 17:07:27.144","2023-08-02 16:07:27,142 DEBUG [] a.a.ActorSystemImpl akka.actor.ActorSystemImpl(ditto-cluster) - http-request: HttpRequest(HttpMethod(POST),http://ingress.companyname.com:8080/api/2/things/com.companyname:secondThingId/features/featureName/inbox/messages/initiateFeatureFlow?timeout=20,List(Timeout-Access: <function1>, Raw-Request-URI: /api/2/things/com.companyname:secondThingId/features/featureName/inbox/messages/initiateFeatureFlow?timeout=20, X-Forwared-User: ditto, x-ditto-pre-authenticated: xxx, Host, X-Real-Ip, X-Forwarded-For, Authorization, User-Agent: PostmanRuntime/7.32.3, Accept: */*, Accept-Encoding: gzip, deflate, br),HttpEntity.Strict(application/json,25 bytes total),HttpProtocol(HTTP/1.1))
Aug 2, 2023 @ 17:07:27.138","2023-08-02 16:07:27,137 DEBUG [] a.a.ActorSystemImpl akka.actor.ActorSystemImpl(ditto-cluster) - http: HttpRequest(HttpMethod(POST),http://ingress.companyname.com:8080/api/2/things/com.companyname:secondThingId/features/featureName/inbox/messages/initiateFeatureFlow?timeout=20,List(Timeout-Access: <function1>, Raw-Request-URI: /api/2/things/com.companyname:secondThingId/features/featureName/inbox/messages/initiateFeatureFlow?timeout=20, X-Forwared-User: ditto, x-ditto-pre-authenticated: xxx, Host, X-Real-Ip, X-Forwarded-For, Authorization, User-Agent: PostmanRuntime/7.32.3, Accept: */*, Accept-Encoding: gzip, deflate, br),HttpEntity.Strict(application/json,25 bytes total),HttpProtocol(HTTP/1.1))

There are no visible WARN/ERROR logs in Ditto related to a specific thing ID, as well as there doesn't seem to be any indication of dropped messages by an mqtt-connection (because the failed case never reaches it as observed in connection logs).

Restart of both devices doesn't change anything. However, there is one extra log statement found related to the failing device:

Aug 2, 2023 @ 17:06:51.921","2023-08-02 16:06:51,916 DEBUG [] a.c.sharding.Shard akka://ditto-cluster@10.250.202.15:2551/system/sharding/search-updater/15 - search-updater: Started entity [Actor[akka://ditto-cluster/system/sharding/search-updater/15/com.companyname%3AsecondThingId#-1090516814]] with entity id [com.companyname:secondThingId] in shard [15]

This seem to state something about shard 15, however all previous and next thingMerged messages (which work fine afaik) are using shard 17. Don't know if that's of any use.

Ditto Version Used: 2.1.2 We do not use Hono, instead relying on self-hosted VerneMQ instance to manage connectivity.

Please assist with troubleshooting this issue.

thjaeckle commented 1 year ago

This is a really ancient Ditto version. A lot has happened since 2.1 and trying to find the issue in that outdated Version will benefit noone.

I would suggest to update Ditto to 3.3.

thjaeckle commented 1 year ago

Closing as outdated issue for a 2 year old Ditto version.