search

espressif / esp-insights

ESP Insights: A remote diagnostics/observability framework for connected devices
Apache License 2.0
101 stars 27 forks source link

Firmware Image Uploaded not Resolving String Information in Error Messages #33

Closed connormartin12 closed 10 months ago

connormartin12 commented 11 months ago

Using the minimal_diagnostics example, I've successfully set up a connection to ESP_Insights. However, when I upload the matching firmware package (I've double checked it's correct according to the ReadMe instructions) it doesn't resolve the string information. I always get the error "We could not find the firmware image to cross-reference this log entry. Please upload a firmware image to get cross-referenced information for this log entry."

For the sake of thoroughness, I've tried this on 2 different esp32s3 modules and one regular esp32 module. Hopefully I'm just an unlucky one as this seems like a fairly big issue.

IDF Version: 5.2 Full Version: v5.2-dev-1890-g28167ea5a3

vikramdattu commented 11 months ago

Hello @connormartin12 the hash of the firmware package has to match to the one which is sending insights from the device. Could it be that the package you uploaded was built after some changes?

Can you check by flashing the firmware on device and uploading it to the insights dashboard from the same build? The older logs still won't be resolved, as they correspond to some other hash. However, newer logs should get resolved.

connormartin12 commented 11 months ago

Hey @vikramdattu, thanks for the prompt reply.

I noticed this was a requirement while reading through the ReadMe file for the example when setting up Insights. However, as a sanity check, I just now ran a fullclean and erase-flash before building and flashing. I uploaded this matching firmware hash to Insights with no change in outcome.

I'll be happy to provide anything else you might need.

vikramdattu commented 11 months ago

@connormartin12 can you please share firmware sha and the node ID for the device?

connormartin12 commented 11 months ago

@vikramdattu Here's the info you requested. I also enabled debug prints in the menuconfig just incase that shows anything insightful. These packages are currently active for this node, however the debug prints enabled sha is the image currently installed on the node.

Node ID: 34851890864C SHA: Attached Insights_Example-v1.zip Insights_Example-v1-debug-prints.zip debug_prints.txt

connormartin12 commented 11 months ago

Hey @vikramdattu any update on the status of this issue?

vikramdattu commented 11 months ago

Hi @connormartin12 we have found the issue with the messages that are sent from your device.

From debug_prints.txt you attached, your device is sending "sha256" : "4f93c257" as sha sum of the firmware, which is just the first half. Insights backend however expects full sha256 of the firmware, which is 4f93c257cb31a4a4 in your case.

My question is, did you make any changes in the firmware which would lead to this? Please check for changes in esp_insights_cbor_encoder.c.

Also, I can see that you are using master branch of the IDF, can you please switch to release/v5.1 and check if the issue persists for the logs with this new firmware?

Update:

@connormartin12 the issue has been identified and will be resolved soon. The issue is limited to IDF master (or v5.2) branch.

Recommend you to use a stable ESP-IDF branch till then for your development., preferably release/v5.1. Please do a clean build (remove previously generated sdkconfig while switching).

connormartin12 commented 11 months ago

@vikramdattu I have confirmed switching to release/v5.1 fixes the issue. Thank you for looking in to this for me!

pablojgauto commented 3 months ago

Hi, I'm using V.5.2.1 and I have the same problem. Shall I switch to release 5.1? Which one do you recommend? Thanks in advance.

connormartin12 commented 3 months ago

Hi @pablojgauto I'm unsure if they've fixed this since I experienced this error. I would recommend switching to the latest stable version of 5.1 and seeing if that solves the issue like it did for me.

pablojgauto commented 3 months ago

Thank you. I'll try it

gadget-man commented 1 month ago

I'm having a similar issue, using ESP-IDF release v5.3 on an esp32s3.

My device is connected to insights, and metrics etc. is all working OK but for every log message I get the error: "We could not find the firmware image to cross-reference this log entry. Please upload a firmware image to get cross-referenced information for this log entry."

In my log output I see on boot that I have ELF file SHA256: 8a776829b..., and in the insights dashboard the uploaded firmware is reporting as 8a776829bfc0f0a1 so I think that's OK. The only error message I get in my logs related to insights is esp_insights: Failed to enable insights_cmd_resp, not sure if that's relevant?

vikramdattu commented 1 month ago

@gadget-man command response should not cause the issue you see. Did you try this suggestion: https://github.com/espressif/esp-insights/issues/33#issuecomment-1744314277 ?

gadget-man commented 1 month ago

Yes I checked that. See my post #45 I've included several screenshots showing exactly the debug log and status in the Insights Dashboard.

vikramdattu commented 1 month ago

I see, can you change the IDF version to release/5.2 or release/5.1 and check if the issue gets resolved. I notice, the issue mentions the IDF version used is 5.3.

If changing IDF version is not an option, setting the config CONFIG_APP_RETRIEVE_LEN_ELF_SHA to 16 (from 9) should fix it.

Please note, every new build with the changes, would require you to upload the package again.

gadget-man commented 1 month ago

Thanks, I can confirm that adding CONFIG_APP_RETRIEVE_LEN_ELF_SHA=16 in 5.3 has resolved the issue.