rFond commented 1 year ago

Hi,

When IOT works for a very long period (60 day) and I launch a reboot with cmd sudo reboot.

IOT can't reboot, with software, I force to do an hard reset electrical. `======================= Board: IOT2050-ADVANCED Serial: xx MLFB: 6ES7647-0BA00-1YA2 UUID: 8069DFE5E3D14C05B9B8D6C4CAC1CEF0 A5E: A5E452229880AB08 MAC[0]: xx MAC[1]: xx SKU: SE Loading PK... ok PK count: 00 PK version: 00 SV: 00-00 Security ID 0x5125e53f-0x842e215d Security policy: soft Loading image atf... Loading image tee... Loading image spl... Loading image k3-am65-iot2050-spl.dtb... NOTICE: BL31: v2.5(release): NOTICE: BL31: Built : 14:41:24, May 17 2021 I/TC: I/TC: OP-TEE version: 3.12.0 (gcc version 10.2.1 20210110 (Debian 10.2.1-6)) #1 Wed Jan 20 17:48:48 UTC 2021 aarch64 I/TC: Primary CPU initializing I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.04-V01.02.01.03-0-g7e29ca7 (Nov 04 2021 - 07:46:39 +0000) SYSFW ABI: 3.1 (firmware rev 0x0015 '21.5.0--v2021.05 (Terrific Llam') Trying to boot from SPI QSPI: QSPI is still busy after poll for 10000 times. SF: Calibration failed (read) cadence_spi spi@47040000: Cannot set speed (err=-5) SPI probe failed. SPL: failed to boot from all boot devices

ERROR ### Please RESET the board

`

Thanks for your help.

rFond commented 1 year ago

All message bugstart_siemens.txt

jan-kiszka commented 1 year ago

Is that error persistent then, or is the error gone again after a cold reset e.g.?

rFond commented 1 year ago

After a cold reset (with button ou electrical), error no gone again.

After.

Board: IOT2050-ADVANCED Serial: NNEM7528 MLFB: 6ES7647-0BA00-1YA2 UUID: 8069DFE5E3D14C05B9B8D6C4CAC1CEF0 A5E: A5E452229880AB08 MAC[0]: xx MAC[1]: xx SKU: SE Loading PK... ok PK count: 00 PK version: 00 SV: 00-00 Security ID 0x5125e53f-0x842e215d Security policy: soft Loading image atf... Loading image tee... Loading image spl... Loading image k3-am65-iot2050-spl.dtb... NOTICE: BL31: v2.5(release): NOTICE: BL31: Built : 14:41:24, May 17 2021 I/TC: I/TC: OP-TEE version: 3.12.0 (gcc version 10.2.1 20210110 (Debian 10.2.1-6)) #1 Wed Jan 20 17:48:48 UTC 2021 aarch64 I/TC: Primary CPU initializing I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.04-V01.02.01.03-0-g7e29ca7 (Nov 04 2021 - 07:46:39 +0000) SYSFW ABI: 3.1 (firmware rev 0x0015 '21.5.0--v2021.05 (Terrific Llam') Trying to boot from SPI

U-Boot 2021.04-V01.02.01.03-0-g7e29ca7 (Nov 04 2021 - 07:46:39 +0000)

Model: SIMATIC IOT2050 Advanced DRAM: 2 GiB WDT: Not starting MMC: sdhci@4f80000: 1, sdhci@4fa0000: 0 Loading Environment from SPIFlash... SF: Detected w25q128 with page size 256 Bytes, erase size 64 KiB, total 16 MiB OK In: serial Out: serial Err: serial Net: No ethernet found. Hit any key to stop autoboot: 0 IOT2050> run bootcmd_mmc1 switch to partitions #0, OK mmc1(part 0) is current device Scanning mmc 1:1... Found U-Boot script /boot/boot.scr 732 bytes read in 9 ms (79.1 KiB/s)

Executing script at 80000000

Loading /usr/lib/linux-image-5.10.162-cip24/ti/k3-am6548-iot2050-advanced.dtb... 49996 bytes read in 13 ms (3.7 MiB/s) Loading /boot/vmlinux-5.10.162-cip24... 18770432 bytes read in 205 ms (87.3 MiB/s)

Flattened Device Tree blob at 88000000

Booting using the fdt blob at 0x88000000 Loading Device Tree to 000000008fff0000, end 000000008ffff34b ... OK

Starting kernel ...

jan-kiszka commented 1 year ago

Ok, at least no bricked devices, still ugly. Seen this once so far only, or is this reproducible? And then really only after 60 days?

@BaochengSu are we aware of any such issue already? Looks to me like imperfect re-initialization of the QSPI controller in U-Boot.

BaochengSu commented 1 year ago

QA reported such issue long before, but this is hard to reproduce then hard to debug... Very likely as you suggested something happened in the u-boot driver.

rFond commented 1 year ago

Yes it's little random, I noticed though if we are greater than 40-60d we have this problem. It bothers me because on the IOT in remote control (vpn or other) I sometimes launch reboots. If IOT no restart I have to call a technician. I try to avoid running reboots.

I can send you my IOT image.

jan-kiszka commented 1 year ago

@rFond are you already on the latest firmware version we released?

BaochengSu commented 1 year ago

@rFond And what is the re-producing rate from your side? If it's still little, then I don't think it will help to have your image.

Anyway, thanks for the reporting and supporting, we will definitely look into this issue from our side.

rFond commented 1 year ago

@jan-kiszka Not yet. I can test no problem. I have a development/test bench.

@BaochengSu Reboot on all my park (50 devices), and device with more than 40-60d operating time. The problem is old because the first version (IOT in debian 10) I have already this problem

BaochengSu commented 1 year ago

@rFond This issue has nothing to do with the image, but the firmware version. The firmware version you are currently running is kind of old, the latest version is 1.3.1.x, you can download the firmware updating package from SIOS page.

Also if possible, you can use our latest master to build the latest firmware to see if such issue is still existing. (Please just ignore the error printing in optee regarding the RPMB, that is a known issue and I am on it now, it should not block you from using it.)

rFond commented 1 year ago

@BaochengSu Yes no problem, for moment I can't download file :

You cannot download this file, since your registration inquiry is still being processed.

Please note: due to the manual review, the registration may take several days.You will get a confirmation e-mail as soon as your download registration is finalized.You can download the software packages with export restrictions only when you have received the positive confirmation.

Thanks for your help.

BaochengSu commented 1 year ago

@BaochengSu Yes no problem, for moment I can't download file :

You cannot download this file, since your registration inquiry is still being processed.

Please note: due to the manual review, the registration may take several days.You will get a confirmation e-mail as soon as your download registration is finalized.You can download the software packages with export restrictions only when you have received the positive confirmation.

Thanks for your help.

@bergmanu , can you comment on this?

rFond commented 1 year ago

@BaochengSu It's ok, I'have upgrade firmware on my test bench. I wait and return to you later.

jan-kiszka commented 1 year ago

Any updates?

rFond commented 1 year ago

Hi, not yet, my test bench was disconnected (electrical) so I couldn't do my test :'(

rFond commented 1 year ago

Hi, I'm come back with a result :

SIMATIC IOT2050 SE-Boot Version: D01.02.02.08-0-gf12580b7-0x0000 BuildDate: 20220728 SYSFW ABI: 3.1 [version: 21] [21.9.1--v2021.09a (Terrific Lla]

Board: IOT2050-ADVANCED Serial: NNEM7528 MLFB: 6ES7647-0BA00-1YA2 UUID: 8069DFE5E3D14C05B9B8D6C4CAC1CEF0 A5E: A5E452229880AB08 MAC[0]: 8c-f3-19-4c-cd-a6 MAC[1]: 8c-f3-19-4c-cd-a5 SKU: SE Loading PK... ok PK count: 00 PK version: 00 SV: 00-00 Security ID 0x5125e53f-0x842e215d Security policy: none No options found, skip. Validating FIT... Loading image atf... Loading image tee... Loading image spl... Loading image k3-am65-iot2050-spl.dtb... NOTICE: BL31: v2.6(release): NOTICE: BL31: Built : 08:03:39, Oct 20 2022 I/TC: I/TC: OP-TEE version: 3.16.0 (gcc version 10.2.1 20210110 (Debian 10.2.1-6)) #1 Thu Oct 20 08:03:39 UTC 2022 aarch64 I/TC: WARNING: This OP-TEE configuration might be insecure! I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html I/TC: Primary CPU initializing I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2022.01-V01.03.01.01-0-gffc3caf (Oct 20 2022 - 08:08:40 +0000) SYSFW ABI: 3.1 (firmware rev 0x0015 '21.9.1--v2021.09a (Terrific Lla') Trying to boot from SPI QSPI: QSPI is still busy after poll for 10000 times. SPI probe failed. SPL: failed to boot from all boot devices

ERROR ### Please RESET the board

With the new firmware V1.3.1 this does not work. I left the IOT running for 60 days without rebooting

rFond commented 1 year ago

After hard reset :

SIMATIC IOT2050 SE-Boot Version: D01.02.02.08-0-gf12580b7-0x0000 BuildDate: 20220728 SYSFW ABI: 3.1 [version: 21] [21.9.1--v2021.09a (Terrific Lla]

Board: IOT2050-ADVANCED Serial: NNEM7528 MLFB: 6ES7647-0BA00-1YA2 UUID: 8069DFE5E3D14C05B9B8D6C4CAC1CEF0 A5E: A5E452229880AB08 MAC[0]: 8c-f3-19-4c-cd-a6 MAC[1]: 8c-f3-19-4c-cd-a5 SKU: SE Loading PK... ok PK count: 00 PK version: 00 SV: 00-00 Security ID 0x5125e53f-0x842e215d Security policy: none No options found, skip. Validating FIT... Loading image atf... Loading image tee... Loading image spl... Loading image k3-am65-iot2050-spl.dtb... NOTICE: BL31: v2.6(release): NOTICE: BL31: Built : 08:03:39, Oct 20 2022 I/TC: I/TC: OP-TEE version: 3.16.0 (gcc version 10.2.1 20210110 (Debian 10.2.1-6)) #1 Thu Oct 20 08:03:39 UTC 2022 aarch64 I/TC: WARNING: This OP-TEE configuration might be insecure! I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html I/TC: Primary CPU initializing I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2022.01-V01.03.01.01-0-gffc3caf (Oct 20 2022 - 08:08:40 +0000) SYSFW ABI: 3.1 (firmware rev 0x0015 '21.9.1--v2021.09a (Terrific Lla') Trying to boot from SPI

U-Boot 2022.01-V01.03.01.01-0-gffc3caf (Oct 20 2022 - 08:08:40 +0000)

Model: SIMATIC IOT2050 Advanced DRAM: 2 GiB WDT: Not starting watchdog@40610000 MMC: mmc@4f80000: 1, mmc@4fa0000: 0 Loading Environment from SPIFlash... SF: Detected w25q128 with page size 256 Bytes, erase size 64 KiB, total 16 MiB OK In: serial Out: serial Err: serial Net: No ethernet found. Hit any key to stop autoboot: 0

jan-kiszka commented 1 year ago

Just to keep you informed: This issue was seen by a second user as well, unfortunately with the same extreme long reproduction times.

adannenb-ti commented 1 year ago

@jan-kiszka, all, For cross-reference reference, this is now also being discussed at https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1274143/am6548-u-boot-spl-could-not-load-u-boot-proper-qspi-qspi-is-still-busy-after-poll-for-10000-times

kogi84 commented 1 year ago

Hallo, we have the same error on many IOTs at various customers, what is realy bad for us und our customers! I did't expect an "### ERROR ### Please RESET the board ###" issue from a Siemens Simatic device!

[86383.521081] reboot: Restarting system

SIMATIC IOT2050 SE-Boot Version: V01.02.01-0-g4524e967-0x0000 BuildDate: 20211221 SYSFW ABI: 3.1 [version: 21] [21.5.0--v2021.05 (Terrific Llam] AVS@[1100 1150 1150]

Board: IOT2050-ADVANCED-PG2 Serial: PODC7888 MLFB: 6ES7647-0BA00-1YA2 UUID: 1A61ACB0F3DE45EAB28B6D4B482EACB2 A5E: A5E520802600AA01 MAC[0]: 8c-f3-19-c5-30-7b MAC[1]: 8c-f3-19-c5-30-7a SKU: SE Loading PK... ok PK count: 00 PK version: 00 SV: 00-00 Security ID 0xc8beb00d-0x7f55c02e Security policy: soft Loading image atf... Loading image tee... Loading image spl... Loading image k3-am65-iot2050-spl.dtb... NOTICE: BL31: v2.5(release): NOTICE: BL31: Built : 14:41:24, May 17 2021 I/TC: I/TC: OP-TEE version: 3.12.0 (gcc version 10.2.1 20210110 (Debian 10.2.1-6)) #1 Wed Jan 20 17:48:48 UTC 2021 aarch64 I/TC: Primary CPU initializing I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.04-V01.02.01-0-g40d3fc0 (Jan 05 2022 - 14:11:27 +0000) SYSFW ABI: 3.1 (firmware rev 0x0015 '21.5.0--v2021.05 (Terrific Llam') Trying to boot from SPI QSPI: QSPI is still busy after poll for 10000 times. SF: Calibration failed (read) cadence_spi spi@47040000: Cannot set speed (err=-5) SPI probe failed. SPL: failed to boot from all boot devices

ERROR ### Please RESET the board

jan-kiszka commented 1 year ago

@kogi84 We are hunting that issue, but it is a very shy beast. So far, we are only aware of cases that happened after months(!) of uptime. Therefore, we are very interesting in learning about more details of the circumstances under which it happened to you. Eventually, we will have to reproduce it much more quickly.

And if it turns out that uptime is a precondition, it could help to have sufficiently frequent scheduled reboots - provided your setup permits those.

kogi84 commented 1 year ago

I have also posted the issue at the Siemens IOT2050 support forum https://support.industry.siemens.com/forum/de/en/posts/iot2050-stat-led-stops-blinking-os-hangs-up/304828/?page=0&pageSize=10#!#post1172396. The IOTs are powered 24/7 but we do periodic reboots, but rebooting causes the error. And yes it seems that the uptime is relevant, in our cases the error usually only occurs after about 50 days of operation. We have many IOTs in use at various customers that have this error, what is realy bad for us und our customers!

jan-kiszka commented 1 year ago

OK, that 50 days is - unfortunately - in line with what we know so far. For that reason, another affected user is now going for a nightly scheduled reboot.

jan-kiszka commented 1 year ago

That said: have you ever experience the case when the uptime was definitely much shorter than 50 days?

kogi84 commented 1 year ago

The shortest time was 27 days, but we don't think that nightly scheduled reboots solve the problem, because we do reboot the IOTs when the maschines are not in use and this should occurs multiple times in this timespan of 27-50 days. Are there reliable results that confirm that the nightly scheduled reboots helps the other affected user to get around the problem?

jan-kiszka commented 1 year ago

We do not yet have a reliable workaround recommendation, still collecting data, in the lab as well as in the field.

kogi84 commented 1 year ago

OK, thank you, I try to collect some more data for you and as workarround we will do the nightly scheduled reboots.

kogi84 commented 1 year ago

Hallo, I had a closer look to our loggings and I am shure, that the nightly scheduled reboots don't solve the problem in our case, because our monitored IOT reboots after 86383 seconds uptime, what is arround 24h, you can see it in my first post:

[86383.521081] reboot: Restarting system_ and runs into the "ERROR ### Please RESET the board". We have arround 40 IOTs at various customers in the field, running in this error arround 50 days, really bad!

kogi84 commented 11 months ago

Hallo, is there any progress regarding the issue and resolution for the problem?

rFond commented 9 months ago

Hello, I'm getting back to you regarding this issue. From what I understand, the issue is hardware-related? So, do we need to occasionally reboot the IoT device via software to resolve this problem? Thank you

huaqianli commented 9 months ago

Hi,

Occasionally rebooting the IoT device may not be able to resolve this issue, as it occurred once during our daily-reboot testing before.

And, after 55 days of daily-reboot testing, I reproduced the similar issue on January 3, 2024, in a version shipped with the following patch:

The captured debug log shows that the error occurred when calling cadence_qspi_apb_command_read with "CQSPI_REG_CONFIG: 80083881" for the first time, as follows:

abnormal register: 
  Source: cadence_qspi_apb_command_read
   CQSPI_REG_CONFIG:              80083881

A bit strange is noted: bit 31 should be 0 in the error case because it is CQSPI_REG_CONFIG_IDLE_LSB, and that is the condition that CQSPI_REG_IS_IDLE is testing for. Otherwise, we can't get more information from this case.

However, this patch had redundant modifications; the latency was reduced a lot, from 5 seconds to 5 milliseconds. This makes it different from the original issue.

I fixed the patch, added more debugging information, and started reproducing on January 16th, but it hasn't been reproduced yet.

cpardotortosa commented 7 months ago

Any advancement?

pscom038 commented 5 months ago

I use IOT2050 firmware V1.3.1 on eMMc found same problem after reset. device run time about 180 days and reset via VPN command line.

Now, is there a way to solve this problem?

kogi84 commented 5 months ago

The issue was opened in May 2023, is there any progress?

jan-kiszka commented 5 months ago

There will be a follow-up soon, but this issue seems to remain nasty.

huaqianli commented 5 months ago

Hi, it is currently recommended to upgrade to version V01.04.01.

pscom038 commented 5 months ago

I see firmware 1.4.1 for IOT2050-SM only. Not sure can it be used with another models?

jan-kiszka commented 5 months ago

Where do you see this? The update package you can fetch from https://support.industry.siemens.com/cs/document/109741799/downloads-for-simatic-iot20x0 is not limited to the SM variant.

kogi84 commented 4 months ago

I see only firmware 1.3.1 on the download page? Where can I find version 1.4.1? IOT2050_FW

rFond commented 4 months ago

I have : Capture d’écran 2024-07-18 140212

And : Firmware-V01.04.zip

kogi84 commented 4 months ago

What, that's strange with the homepage link, have try it on several browsers, but I see an old version of the page. Thank you for the Firmware!

jan-kiszka commented 4 months ago

I am confused myself, only seeing 1.3.1 right now as well. Let's check back what is going on there.

pscom038 commented 4 months ago

I have installed firmware and image version 1.4.1 on about 10 devices. Hopefully this problem has been fixed. Thk. Capture

uwedaeu commented 4 months ago

Right now we are facing some problems with the internal translation. To get access to the FW 1.4.1 go to https://support.industry.siemens.com/cs/document/109741799/downloads-for-simatic-iot20x0 and switch your language to english the FW version 1.4.1 should now be visible:

rFond commented 3 months ago

Hi ! For information after 51 days with new firmware 1.04 same issue :

siemens@iot2050-debian:~$ uptime
 10:30:07 up 51 days, 20:28,  3 users,  load average: 2.48, 1.08, 0.93
siemens@iot2050-debian:~$ sudo reboot

[  OK  ] Unmounted /run/docker/netns/c53b4e6dd2c3.
[  OK  ] Unmounted /var/lib/docker/…832ec2a7688e27867e8987/merged.
[  OK  ] Stopped Docker Application Container Engine.
[  OK  ] Stopped target Network is Online.
[  OK  ] Stopped Network Manager Wait Online.
         Stopping containerd container runtime...
[  OK  ] Stopped containerd container runtime.
[  OK  ] Stopped target Network.
         Stopping Network Manager...
         Stopping Raise network interfaces...
         Stopping WPA supplicant...
[  OK  ] Stopped Raise network interfaces.
[  OK  ] Stopped WPA supplicant.
[  OK  ] Stopped Network Manager.
[  OK  ] Stopped target Network (Pre).
         Stopping D-Bus System Message Bus...
[  OK  ] Stopped D-Bus System Message Bus.
[  OK  ] Stopped target Basic System.
[  OK  ] Stopped target Paths.
[  OK  ] Stopped target Slices.
[  OK  ] Removed slice User and Session Slice.
[  OK  ] Stopped target Sockets.
[  OK  ] Closed D-Bus System Message Bus Socket.
[  OK  ] Closed Docker Socket for the API.
[  OK  ] Closed Syslog Socket.
[  OK  ] Closed UUID daemon activation socket.
[  OK  ] Stopped target System Initialization.
[  OK  ] Stopped target Local Encrypted Volumes.
[  OK  ] Stopped Dispatch Password …ts to Console Directory Watch.
[  OK  ] Stopped Forward Password R…uests to Wall Directory Watch.
[  OK  ] Stopped target Swap.
[  OK  ] Stopped Apply Kernel Variables.
[  OK  ] Stopped Load Kernel Modules.
         Stopping Update UTMP about System Boot/Shutdown...
[  OK  ] Stopped Update UTMP about System Boot/Shutdown.
[  OK  ] Stopped Create Volatile Files and Directories.
[  OK  ] Stopped target Local File Systems.
         Unmounting /run/docker/netns/54163bec4b0c...
         Unmounting /var/lib/docker…75389e775e13b69e3005/merged...
[  OK  ] Unmounted /run/docker/netns/54163be[4480117.790597] docker1: port 9(veth9d26656) entered disabled state
c4b0c.
[4480117.807867] device veth9d26656 left promiscuous mode
[4480117.815166] docker1: port 9(veth9d26656) entered disabled state
[  OK  ] Unmounted /var/lib/docker/…5d75389e775e13b69e3005/merged.
[  OK  ] Stopped target Local File Systems (Pre).
[  OK  ] Reached target Unmount All Filesystems.
[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Create System Users.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Reached target Shutdown.
[  OK  ] Reached target Final Step.
[  OK  ] Finished Reboot.
[  OK  ] Reached target Reboot.
[4480118.129225] rti-wdt 40610000.watchdog: new timeout value does not match already configured one
[4480118.140393] watchdog: watchdog0: nowayout prevents watchdog being stopped!
[4480118.147526] watchdog: watchdog0: watchdog did not stop!
[4480118.264270] systemd-shutdown[1]: Syncing filesystems and block devices.
[4480118.300673] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[4480118.331392] systemd-journald[184]: Received SIGTERM from PID 1 (systemd-shutdow).
[4480118.366411] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[4480118.387780] systemd-shutdown[1]: Using hardware watchdog 'K3 RTI Watchdog', version 0, device /dev/watchdog
[4480118.402641] systemd-shutdown[1]: Unmounting file systems.
[4480118.414187] [1835198]: Remounting '/' read-only in with options '(null)'.
[4480118.615520] EXT4-fs (mmcblk1p1): re-mounted. Opts: (null)
[4480118.671592] systemd-shutdown[1]: All filesystems unmounted.
[4480118.677474] systemd-shutdown[1]: Deactivating swaps.
[4480118.682961] systemd-shutdown[1]: All swaps deactivated.
[4480118.688431] systemd-shutdown[1]: Detaching loop devices.
[4480118.698676] systemd-shutdown[1]: All loop devices detached.
[4480118.704505] systemd-shutdown[1]: Stopping MD devices.
[4480118.710335] systemd-shutdown[1]: All MD devices stopped.
[4480118.715945] systemd-shutdown[1]: Detaching DM devices.
[4480118.721950] systemd-shutdown[1]: All DM devices detached.
[4480118.727780] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[4480118.755138] systemd-shutdown[1]: Syncing filesystems and block devices.
[4480118.762255] systemd-shutdown[1]: Rebooting.
[4480118.787802] reboot: Restarting system
AVS@[1100 1190 1190]
SIMATIC IOT2050 SE-Boot Version: V01.04.01-0-g629b172b-0x0000
BuildDate: 20240103
SYSFW ABI: 3.1 [version: 21] [21.9.1--v2021.09a (Terrific Lla]
=======================
Board: IOT2050-ADVANCED
Serial: NNEM7528
MLFB: 6ES7647-0BA00-1YA2
UUID: 8069DFE5E3D14C05B9B8D6C4CAC1CEF0
A5E: A5E452229880AB08
MAC[0]: 8c-f3-19-4c-cd-a6
MAC[1]: 8c-f3-19-4c-cd-a5
DDR: 2048MB
SKU: SE
Loading PK... ok
PK count: 00
PK version: 00
SV: 00-00
Security ID 0x5125e53f-0x842e215d
Security policy: none
No options found, skip.
Validating FIT...
Loading image atf...
Loading image tee...
Loading image spl...
Loading image fdt-0...
NOTICE:  BL31: lts-v2.8.10(release):
NOTICE:  BL31: Built : 01:00:00, Jan  1 1970
I/TC:
I/TC: OP-TEE version: 4.0.0 (gcc version 12.2.0 (Debian 12.2.0-14)) #1 Thu Jan  1 01:00:00 UTC 1970 aarch64
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0015 '21.9.1--v2021.09a (Terrific Lla')
I/TC: Secure Board Configuration Software: Rev 0
I/TC: Secure Boot Keys: Count 2, Rev 1
I/TC: Activated SA2UL device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2023.10-V01.04.01-0-gb5963ab-dirty (Jan 01 1970 - 01:00:00 +0000)
SYSFW ABI: 3.1 (firmware rev 0x0015 '21.9.1--v2021.09a (Terrific Lla')
Trying to boot from SPI
QSPI: QSPI is still busy after poll for 10000 times.
SPI probe failed.
SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###

rFond commented 3 months ago

After hard reset :

AVS@[1100 1190 1190]
SIMATIC IOT2050 SE-Boot Version: V01.04.01-0-g629b172b-0x0000
BuildDate: 20240103
SYSFW ABI: 3.1 [version: 21] [21.9.1--v2021.09a (Terrific Lla]
=======================
Board: IOT2050-ADVANCED
Serial: NNEM7528
MLFB: 6ES7647-0BA00-1YA2
UUID: 8069DFE5E3D14C05B9B8D6C4CAC1CEF0
A5E: A5E452229880AB08
MAC[0]: 8c-f3-19-4c-cd-a6
MAC[1]: 8c-f3-19-4c-cd-a5
DDR: 2048MB
SKU: SE
Loading PK... ok
PK count: 00
PK version: 00
SV: 00-00
Security ID 0x5125e53f-0x842e215d
Security policy: none
No options found, skip.
Validating FIT...
Loading image atf...
Loading image tee...
Loading image spl...
Loading image fdt-0...
NOTICE:  BL31: lts-v2.8.10(release):
NOTICE:  BL31: Built : 01:00:00, Jan  1 1970
I/TC:
I/TC: OP-TEE version: 4.0.0 (gcc version 12.2.0 (Debian 12.2.0-14)) #1 Thu Jan  1 01:00:00 UTC 1970 aarch64
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0015 '21.9.1--v2021.09a (Terrific Lla')
I/TC: Secure Board Configuration Software: Rev 0
I/TC: Secure Boot Keys: Count 2, Rev 1
I/TC: Activated SA2UL device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2023.10-V01.04.01-0-gb5963ab-dirty (Jan 01 1970 - 01:00:00 +0000)
SYSFW ABI: 3.1 (firmware rev 0x0015 '21.9.1--v2021.09a (Terrific Lla')
Trying to boot from SPI

U-Boot 2023.10-V01.04.01-0-gb5963ab-dirty (Jan 01 1970 - 01:00:00 +0000)

Model: SIMATIC IOT2050 Advanced
DRAM:  2 GiB
Core:  176 devices, 32 uclasses, devicetree: separate
WDT:   Not starting watchdog@40610000
MMC:   mmc@4f80000: 1, mmc@4fa0000: 0
Loading Environment from SPIFlash... SF: Detected w25q128 with page size 256 Bytes, erase size 64 KiB, total 16 MiB
OK
In:    serial@2810000
Out:   serial@2810000
Err:   serial@2810000
Net:   No ethernet found.
Hit SPACE to stop autoboot in 2 seconds...
MMC: no card present
switch to partitions #0, OK
mmc1(part 0) is current device
Scanning mmc 1:1...
Found U-Boot script /boot/boot.scr
732 bytes read in 30 ms (23.4 KiB/s)
## Executing script at 80000000
Loading /usr/lib/linux-image-5.10.162-cip24/ti/k3-am6548-iot2050-advanced.dtb...
56965 bytes read in 34 ms (1.6 MiB/s)
Loading /boot/vmlinux-5.10.162-cip24...
18842112 bytes read in 230 ms (78.1 MiB/s)
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
Working FDT set to 88000000
   Loading Device Tree to 000000008ffef000, end 000000008ffffe84 ... OK
Working FDT set to 8ffef000

Starting kernel ...

rFond commented 1 month ago

some news ?

huaqianli commented 1 month ago

Hi @rFond ,

Thanks for your information.

We have been testing a debug firmware for over a month without any failures so far. We plan to release it early next month.

Details:

The original QSPI issue has been fixed with our workaround.
However, after resolving the QSPI issue, the device fails to boot with eMMC. We have added a potential workaround for the eMMC issue:
- TI provided a workaround to enable the MMC reset function.
- We added a workaround to reset the eMMC.

Example Scenario:

The device runs for 2 months.
The device fails to boot due to the QSPI issue. (If the device is running with the QSPI workaround, this issue will be bypassed and proceed to step 3.)
The device fails to boot due to not being able to detect the eMMC. (The symptom is the same as the QSPI issue, where the device fails to boot and can only be recovered with a cold reset.)

We now have a potential workaround for step 3 (the eMMC issue). Our testing is not yet complete, but it should improve the situation.

Thanks.

uwedaeu commented 2 weeks ago

Hello all, the firmware version V1.4.3 with the workaround for this issue was published today on our Downloads Page.

Further information can be found in our new forum entry.

Thanks

siemens / meta-iot2050

Reboot problem #440

ERROR ### Please RESET the board

After.

Executing script at 80000000

Flattened Device Tree blob at 88000000

You cannot download this file, since your registration inquiry is still being processed.

You cannot download this file, since your registration inquiry is still being processed.

SIMATIC IOT2050 SE-Boot Version: D01.02.02.08-0-gf12580b7-0x0000 BuildDate: 20220728 SYSFW ABI: 3.1 [version: 21] [21.9.1--v2021.09a (Terrific Lla]

ERROR ### Please RESET the board

SIMATIC IOT2050 SE-Boot Version: D01.02.02.08-0-gf12580b7-0x0000 BuildDate: 20220728 SYSFW ABI: 3.1 [version: 21] [21.9.1--v2021.09a (Terrific Lla]

SIMATIC IOT2050 SE-Boot Version: V01.02.01-0-g4524e967-0x0000 BuildDate: 20211221 SYSFW ABI: 3.1 [version: 21] [21.5.0--v2021.05 (Terrific Llam] AVS@[1100 1150 1150]

ERROR ### Please RESET the board