search

OP-TEE / optee_os

Trusted side of the TEE
Other
1.59k stars 1.07k forks source link

rcu_preempt stalls detected on EMSG, IMSG and DMSG in optee #6978

Closed Muhammadhassan1234 closed 1 month ago

Muhammadhassan1234 commented 3 months ago

Hi Community!

I have just started working on optee and found out that whenever logs like EMSG; IMSG and DMSG comes in my application, my system gots hanged and gives the following error

rcu: INFO: rcu_preempt detected stalls on CP/1/0x4000000000000000 softirq=11458/11459 fqs=1051

When I removed such logs from the Trusted application (TA), my optee examples work fine. Does Anyone have any idea how to resolve this error?

Here are the different things that I tried to resolve this error

  1. I have already played with CFG_TEE_TA_LOG_LEVELand CFG_TEE_CORE_LOG_LEVEL. I have tried different log levels but it didn't work.
  2. After that, I have also tried to change the debug UART of optee but it still hangs.
  3. I also increased the rcu_preempt time but it didn't work too
  4. I also tried to read code where it was mentioned that if optee core log level is less than some minimum level, these EMSG, IMSG and DMSG won't take effect. I did it too but it still hanged. One more interesting thing that I noticed that these EMSG, DMSG and IMSG logs are also used in optee uart driver even when UART are not initialized. Where these logs dump the data. Any idea about this? If I am percieving it wrong please let me know!
jenswi-linaro commented 3 months ago

It sounds like the CPU might hang when you're logging from OP-TEE. There's a spinlock involved so if one CPU crashes after taking the spinlock it will block all other logging.

Muhammadhassan1234 commented 3 months ago

@jenswi-linaro how can I deal with spin lock issue?

jenswi-linaro commented 3 months ago

We don't know if there is a spinlock issue yet. If you can attach a debugger it should be easy to see where it's stuck. But if you can't do that on your platform then you're left with trying to debug this with debug prints. I can't tell you where to add the prints to narrow down the problem. If you suspect the spinlock you could try to disable it in trace_ext_puts() since it's only used to avoid interleaved logging.

Muhammadhassan1234 commented 3 months ago

@jenswi-linaro Thanks for the reply. I will try to disable it in trace_ext_puts() as I can't use the debugger right now.

github-actions[bot] commented 2 months ago

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment, otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.