Strange interrupt behavior

[i-nevo]

Hi, I recently encountered a problem when handling secure interrupts. The problematic scenario is as follow:

• There are 2 periodical timers triggering secure interrupts, one is defined as group 0 (EL3) and one as group 1-S (EL1-S).
• The group 1s interrupt has the higher priority and high frequency relative to the group 0 interrupt.
• OPTEE has a handler for handling the group 1s interrupt, ATF has a handler for the group 0 interrupt.
• OPTEE receives a request to load & run a new TA, during this process all the group 1s interrupts are received as IRQ (native).
• In rare cases, the handling of the group 1s interrupt (as IRQ) takes more time than usual, and during the handling both timers expire.
• After handling is complete, in routine gic_it_handle an EOI is sent to GIC.
• GIC immediately sends interrupt signal for the group 0 interrupt.
• Since FIQ is not masked in IRQ handling, ATF gets FIQ exception exiting the IRQ handling before it is finished.
• ATF reads the highest priority pending interrupt, which is the pending group 1s interrupt and exits to the OPTEE FIQ entry-point.
• OPTEE starts handling an interrupt before completing the handling of the last interrupt and fails resuming the TA loading.

I managed to solve the problem in 2 ways: Masking Foreign interrupts (FIQ) before exiting the registered handler of the group 1s interrupt or by making sure that all group 0 interrupts have higher priorities than all group 1s interrupts. However, it seems to me like a problem in OPTEE that such scenario can happen where the handling of an interrupt can start while handling an older interrupt isn't finished.

Regards, Izhar Nevo.

[jenswi-linaro]

So you have EL3 interrupts with lower priority than the S-EL1 interrupts? Is there a way for TF-A to exclude S-EL1 interrupts when the interrupt handler is entered from the secure world?

[i-nevo]

Yes, that's the interrupts configuration. In the TF-A OPTEE Dispatcher, there's an assert in the S-EL1 interrupts handler that is supposed to crash if the S-EL1 interrupt was originated in the Secure world. In our TF-A all asserts are empty macros, so it does nothing, but if it did, we would have crashed anyway.

[jenswi-linaro]

Is it possible to use a higher priority for the EL3 interrupts?

[i-nevo]

Of course it's possible and I mentioned this in the original post as one of the two solutions we found for the problem (we actually decided to use the other one), but it restricts interrupt handling.

[jenswi-linaro]

It does, I guess you use the EL3 interrupts for a reason. Changing the priorities should avoid that. How would you expect OP-TEE to work around this?

[i-nevo]

The EL3 interrupt is a low-frequency periodic timer interrupt for system monitoring, so it's low-priority. The S-EL1 interrupts are HW events requiring fast response with low latency, so they are high-priority. I think that OPTEE should do something similar to what I did, but more subtle: Before writing to GIC EIOR, mask both foreign and native interrupts. From this point it's a very short time (much less than a micro-second) before reaching the end of the interrupt handler where ERET will restore the masks prior to the interrupt (or SMC will return to ATF).

[jenswi-linaro]

I see, feel free to propose that in a pull request.

[github-actions[bot]]

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment, otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.

[jenswi-linaro]

Is this still a problem?

[github-actions[bot]]

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment, otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.