search

OP-TEE / optee_os

Trusted side of the TEE
Other
1.59k stars 1.07k forks source link

Questions about TA preemption behavior #7087

Closed bstandaert-wustl closed 3 weeks ago

bstandaert-wustl commented 3 weeks ago

Hello, as part of a research project I’m doing into scheduling of TAs, I’m trying to understand when OP-TEE allows long-running TAs to be preempted.

From the discussion in #4781, I understood that TAs could only be interrupted if they called Wait(); otherwise they would execute in a single non-preemptive chunk. However, I put together a testcase, where that seems to not be the case: https://gist.github.com/bstandaert-wustl/6f2c7a7f53c4c48363d1831246745887

In my testcase, I create one subthread that I pin to core 1, which invokes a TA that spins for 20 seconds using the TA persistent time APIs. Meanwhile, I repeatedly create other child threads at a 100ms interval, each of which I also pin to core 1; these threads print a message and increment a counter when they execute. Both types of threads are set to use the FIFO scheduler; I give the thread running the TA the lowest priority of the group, so that if the scheduler is picking a new task, it will always pick one of the message-printing threads.

If my TA was nonpreemptible, I would expect it to block other tasks from starting on the core while it executed, but what I actually see happen is that the other tasks, on the same CPU core as the TA thread, continuously launch and execute at the same time as the TA, as shown in the collapsed log below:

Welcome to Buildroot, type root or test to login
buildroot login: root
# 
# echo -1 > /proc/sys/kernel/sched_rt_runtime_us
# optee_example_hello_world_testing
controller on core 0
TA thread on cpu 1
invoking TEE spin command
self context switches: 12 vol, 83 invol
current thread context switches 0 vol, 0 invol
timer count 1; exec count 0
timer task initially on cpu 1
timer count 2; exec count 1
timer task initially on cpu 1
timer count 3; exec count 2
timer task initially on cpu 1
timer count 4; exec count 3
timer task initially on cpu 1
timer count 5; exec count 4
timer task initially on cpu 1
timer count 6; exec count 5
timer task initially on cpu 1
timer count 7; exec count 6
timer task initially on cpu 1
timer count 8; exec count 7
timer task initially on cpu 1
timer count 9; exec count 8
timer task initially on cpu 1
timer count 10; exec count 9
timer task initially on cpu 1
timer count 11; exec count 10
timer task initially on cpu 1
timer count 12; exec count 11
timer task initially on cpu 1
timer count 13; exec count 12
timer task initially on cpu 1
timer count 14; exec count 13
timer task initially on cpu 1
timer count 15; exec count 14
timer task initially on cpu 1
timer count 16; exec count 15
timer task initially on cpu 1
timer count 17; exec count 16
timer task initially on cpu 1
timer count 18; exec count 17
timer task initially on cpu 1
timer count 19; exec count 18
timer task initially on cpu 1
timer count 20; exec count 19
timer task initially on cpu 1
timer count 21; exec count 20
timer task initially on cpu 1
timer count 22; exec count 21
timer task initially on cpu 1
timer count 23; exec count 22
timer task initially on cpu 1
timer count 24; exec count 23
timer task initially on cpu 1
timer count 25; exec count 24
timer task initially on cpu 1
timer count 26; exec count 25
timer task initially on cpu 1
timer count 27; exec count 26
timer task initially on cpu 1
timer count 28; exec count 27
timer task initially on cpu 1
timer count 29; exec count 28
timer task initially on cpu 1
timer count 30; exec count 29
timer task initially on cpu 1
timer count 31; exec count 30
timer task initially on cpu 1
timer count 32; exec count 31
timer task initially on cpu 1
timer count 33; exec count 32
timer task initially on cpu 1
timer count 34; exec count 33
timer task initially on cpu 1
timer count 35; exec count 34
timer task initially on cpu 1
timer count 36; exec count 35
timer task initially on cpu 1
timer count 37; exec count 36
timer task initially on cpu 1
timer count 38; exec count 37
timer task initially on cpu 1
timer count 39; exec count 38
timer task initially on cpu 1
timer count 40; exec count 39
timer task initially on cpu 1
timer count 41; exec count 40
timer task initially on cpu 1
timer count 42; exec count 41
timer task initially on cpu 1
timer count 43; exec count 42
timer task initially on cpu 1
timer count 44; exec count 43
timer task initially on cpu 1
timer count 45; exec count 44
timer task initially on cpu 1
timer count 46; exec count 45
timer task initially on cpu 1
timer count 47; exec count 46
timer task initially on cpu 1
timer count 48; exec count 47
timer task initially on cpu 1
timer count 49; exec count 48
timer task initially on cpu 1
timer count 50; exec count 49
timer task initially on cpu 1
timer count 51; exec count 50
timer task initially on cpu 1
timer count 52; exec count 51
timer task initially on cpu 1
timer count 53; exec count 52
timer task initially on cpu 1
timer count 54; exec count 53
timer task initially on cpu 1
timer count 55; exec count 54
timer task initially on cpu 1
timer count 56; exec count 55
timer task initially on cpu 1
timer count 57; exec count 56
timer task initially on cpu 1
timer count 58; exec count 57
timer task initially on cpu 1
timer count 59; exec count 58
timer task initially on cpu 1
timer count 60; exec count 59
timer task initially on cpu 1
timer count 61; exec count 60
timer task initially on cpu 1
timer count 62; exec count 61
timer task initially on cpu 1
timer count 63; exec count 62
timer task initially on cpu 1
timer count 64; exec count 63
timer task initially on cpu 1
timer count 65; exec count 64
timer task initially on cpu 1
timer count 66; exec count 65
timer task initially on cpu 1
timer count 67; exec count 66
timer task initially on cpu 1
timer count 68; exec count 67
timer task initially on cpu 1
timer count 69; exec count 68
timer task initially on cpu 1
timer count 70; exec count 69
timer task initially on cpu 1
timer count 71; exec count 70
timer task initially on cpu 1
timer count 72; exec count 71
timer task initially on cpu 1
timer count 73; exec count 72
timer task initially on cpu 1
timer count 74; exec count 73
timer task initially on cpu 1
timer count 75; exec count 74
timer task initially on cpu 1
timer count 76; exec count 75
timer task initially on cpu 1
timer count 77; exec count 76
timer task initially on cpu 1
timer count 78; exec count 77
timer task initially on cpu 1
timer count 79; exec count 78
timer task initially on cpu 1
timer count 80; exec count 79
timer task initially on cpu 1
timer count 81; exec count 80
timer task initially on cpu 1
timer count 82; exec count 81
timer task initially on cpu 1
timer count 83; exec count 82
timer task initially on cpu 1
timer count 84; exec count 83
timer task initially on cpu 1
timer count 85; exec count 84
timer task initially on cpu 1
timer count 86; exec count 85
timer task initially on cpu 1
timer count 87; exec count 86
timer task initially on cpu 1
timer count 88; exec count 87
timer task initially on cpu 1
timer count 89; exec count 88
timer task initially on cpu 1
timer count 90; exec count 89
timer task initially on cpu 1
timer count 91; exec count 90
timer task initially on cpu 1
timer count 92; exec count 91
timer task initially on cpu 1
timer count 93; exec count 92
timer task initially on cpu 1
timer count 94; exec count 93
timer task initially on cpu 1
timer count 95; exec count 94
timer task initially on cpu 1
timer count 96; exec count 95
timer task initially on cpu 1
timer count 97; exec count 96
timer task initially on cpu 1
timer count 98; exec count 97
timer task initially on cpu 1
timer count 99; exec count 98
timer task initially on cpu 1
timer count 100; exec count 99
timer task initially on cpu 1
timer count 101; exec count 100
timer task initially on cpu 1
timer count 102; exec count 101
timer task initially on cpu 1
timer count 103; exec count 102
timer task initially on cpu 1
timer count 104; exec count 103
timer task initially on cpu 1
timer count 105; exec count 104
timer task initially on cpu 1
timer count 106; exec count 105
timer task initially on cpu 1
timer count 107; exec count 106
timer task initially on cpu 1
timer count 108; exec count 107
timer task initially on cpu 1
timer count 109; exec count 108
timer task initially on cpu 1
timer count 110; exec count 109
timer task initially on cpu 1
timer count 111; exec count 110
timer task initially on cpu 1
timer count 112; exec count 111
timer task initially on cpu 1
timer count 113; exec count 112
timer task initially on cpu 1
timer count 114; exec count 113
timer task initially on cpu 1
timer count 115; exec count 114
timer task initially on cpu 1
timer count 116; exec count 115
timer task initially on cpu 1
timer count 117; exec count 116
timer task initially on cpu 1
timer count 118; exec count 117
timer task initially on cpu 1
timer count 119; exec count 118
timer task initially on cpu 1
timer count 120; exec count 119
timer task initially on cpu 1
timer count 121; exec count 120
timer task initially on cpu 1
timer count 122; exec count 121
timer task initially on cpu 1
timer count 123; exec count 122
timer task initially on cpu 1
timer count 124; exec count 123
timer task initially on cpu 1
timer count 125; exec count 124
timer task initially on cpu 1
timer count 126; exec count 125
timer task initially on cpu 1
timer count 127; exec count 126
timer task initially on cpu 1
timer count 128; exec count 127
timer task initially on cpu 1
timer count 129; exec count 128
timer task initially on cpu 1
timer count 130; exec count 129
timer task initially on cpu 1
timer count 131; exec count 130
timer task initially on cpu 1
timer count 132; exec count 131
timer task initially on cpu 1
timer count 133; exec count 132
timer task initially on cpu 1
timer count 134; exec count 133
timer task initially on cpu 1
timer count 135; exec count 134
timer task initially on cpu 1
timer count 136; exec count 135
timer task initially on cpu 1
timer count 137; exec count 136
timer task initially on cpu 1
timer count 138; exec count 137
timer task initially on cpu 1
timer count 139; exec count 138
timer task initially on cpu 1
timer count 140; exec count 139
timer task initially on cpu 1
timer count 141; exec count 140
timer task initially on cpu 1
timer count 142; exec count 141
timer task initially on cpu 1
timer count 143; exec count 142
timer task initially on cpu 1
timer count 144; exec count 143
timer task initially on cpu 1
timer count 145; exec count 144
timer task initially on cpu 1
timer count 146; exec count 145
timer task initially on cpu 1
timer count 147; exec count 146
timer task initially on cpu 1
timer count 148; exec count 147
timer task initially on cpu 1
timer count 149; exec count 148
timer task initially on cpu 1
timer count 150; exec count 149
timer task initially on cpu 1
timer count 151; exec count 150
timer task initially on cpu 1
timer count 152; exec count 151
timer task initially on cpu 1
timer count 153; exec count 152
timer task initially on cpu 1
timer count 154; exec count 153
timer task initially on cpu 1
timer count 155; exec count 154
timer task initially on cpu 1
timer count 156; exec count 155
timer task initially on cpu 1
timer count 157; exec count 156
timer task initially on cpu 1
timer count 158; exec count 157
timer task initially on cpu 1
timer count 159; exec count 158
timer task initially on cpu 1
timer count 160; exec count 159
timer task initially on cpu 1
timer count 161; exec count 160
timer task initially on cpu 1
timer count 162; exec count 161
timer task initially on cpu 1
timer count 163; exec count 162
timer task initially on cpu 1
timer count 164; exec count 163
timer task initially on cpu 1
timer count 165; exec count 164
timer task initially on cpu 1
timer count 166; exec count 165
timer task initially on cpu 1
timer count 167; exec count 166
timer task initially on cpu 1
timer count 168; exec count 167
timer task initially on cpu 1
timer count 169; exec count 168
timer task initially on cpu 1
timer count 170; exec count 169
timer task initially on cpu 1
timer count 171; exec count 170
timer task initially on cpu 1
timer count 172; exec count 171
timer task initially on cpu 1
timer count 173; exec count 172
timer task initially on cpu 1
timer count 174; exec count 173
timer task initially on cpu 1
timer count 175; exec count 174
timer task initially on cpu 1
timer count 176; exec count 175
timer task initially on cpu 1
timer count 177; exec count 176
timer task initially on cpu 1
timer count 178; exec count 177
timer task initially on cpu 1
timer count 179; exec count 178
timer task initially on cpu 1
timer count 180; exec count 179
timer task initially on cpu 1
timer count 181; exec count 180
timer task initially on cpu 1
timer count 182; exec count 181
timer task initially on cpu 1
timer count 183; exec count 182
timer task initially on cpu 1
timer count 184; exec count 183
timer task initially on cpu 1
timer count 185; exec count 184
timer task initially on cpu 1
timer count 186; exec count 185
timer task initially on cpu 1
timer count 187; exec count 186
timer task initially on cpu 1
timer count 188; exec count 187
timer task initially on cpu 1
timer count 189; exec count 188
timer task initially on cpu 1
timer count 190; exec count 189
timer task initially on cpu 1
timer count 191; exec count 190
timer task initially on cpu 1
timer count 192; exec count 191
timer task initially on cpu 1
timer count 193; exec count 192
timer task initially on cpu 1
timer count 194; exec count 193
timer task initially on cpu 1
timer count 195; exec count 194
timer task initially on cpu 1
timer count 196; exec count 195
timer task initially on cpu 1
timer count 197; exec count 196
timer task initially on cpu 1
timer count 198; exec count 197
timer task initially on cpu 1
timer count 199; exec count 198
timer task initially on cpu 1
timer count 200; exec count 199
timer task initially on cpu 1
self context switches: 12 vol, 2551 invol
current thread context switches 0 vol, 200 invol
elapsed time 20.021 sec
TEE spin command done
task exec count after spin 200
timer count 201; exec count 200
timer task initially on cpu 1
timer count 202; exec count 201
timer task initially on cpu 1

If I turn off foreign interrupts here, then execution of my other tasks are deferred until the TA returns: https://github.com/OP-TEE/optee_os/blob/master/core/tee/entry_std.c#L538, which produces the following log:

buildroot login: root
# echo -1 > /proc/sys/kernel/sched_rt_runtime_us
# optee_example_hello_world_testing
controller on core 0
TA thread on cpu 1
invoking TEE spin command
self context switches: 20 vol, 72 invol
current thread context switches 0 vol, 0 invol
timer count 1; exec count 0
timer count 2; exec count 0
timer count 3; exec count 0
timer count 4; exec count 0
timer count 5; exec count 0
timer count 6; exec count 0
timer count 7; exec count 0
timer count 8; exec count 0
timer count 9; exec count 0
timer count 10; exec count 0
timer count 11; exec count 0
timer count 12; exec count 0
timer count 13; exec count 0
timer count 14; exec count 0
timer count 15; exec count 0
timer count 16; exec count 0
timer count 17; exec count 0
timer count 18; exec count 0
timer count 19; exec count 0
timer count 20; exec count 0
timer count 21; exec count 0
timer count 22; exec count 0
timer count 23; exec count 0
timer count 24; exec count 0
timer count 25; exec count 0
timer count 26; exec count 0
timer count 27; exec count 0
timer count 28; exec count 0
timer count 29; exec count 0
timer count 30; exec count 0
timer count 31; exec count 0
timer count 32; exec count 0
timer count 33; exec count 0
timer count 34; exec count 0
timer count 35; exec count 0
timer count 36; exec count 0
timer count 37; exec count 0
timer count 38; exec count 0
timer count 39; exec count 0
timer count 40; exec count 0
timer count 41; exec count 0
timer count 42; exec count 0
timer count 43; exec count 0
timer count 44; exec count 0
timer count 45; exec count 0
timer count 46; exec count 0
timer count 47; exec count 0
timer count 48; exec count 0
timer count 49; exec count 0
timer count 50; exec count 0
timer count 51; exec count 0
timer count 52; exec count 0
timer count 53; exec count 0
timer count 54; exec count 0
timer count 55; exec count 0
timer count 56; exec count 0
timer count 57; exec count 0
timer count 58; exec count 0
timer count 59; exec count 0
timer count 60; exec count 0
timer count 61; exec count 0
timer count 62; exec count 0
timer count 63; exec count 0
timer count 64; exec count 0
timer count 65; exec count 0
timer count 66; exec count 0
timer count 67; exec count 0
timer count 68; exec count 0
timer count 69; exec count 0
timer count 70; exec count 0
timer count 71; exec count 0
timer count 72; exec count 0
timer count 73; exec count 0
timer count 74; exec count 0
timer count 75; exec count 0
timer count 76; exec count 0
timer count 77; exec count 0
timer count 78; exec count 0
timer count 79; exec count 0
timer count 80; exec count 0
timer count 81; exec count 0
timer count 82; exec count 0
timer count 83; exec count 0
timer count 84; exec count 0
timer count 85; exec count 0
timer count 86; exec count 0
timer count 87; exec count 0
timer count 88; exec count 0
timer count 89; exec count 0
timer count 90; exec count 0
timer count 91; exec count 0
timer count 92; exec count 0
timer count 93; exec count 0
timer count 94; exec count 0
timer count 95; exec count 0
timer count 96; exec count 0
timer count 97; exec count 0
timer count 98; exec count 0
timer count 99; exec count 0
timer count 100; exec count 0
timer count 101; exec count 0
timer count 102; exec count 0
timer count 103; exec count 0
timer count 104; exec count 0
timer count 105; exec count 0
timer count 106; exec count 0
timer count 107; exec count 0
timer count 108; exec count 0
timer count 109; exec count 0
timer count 110; exec count 0
timer count 111; exec count 0
timer count 112; exec count 0
timer count 113; exec count 0
timer count 114; exec count 0
timer count 115; exec count 0
timer count 116; exec count 0
timer count 117; exec count 0
timer count 118; exec count 0
timer count 119; exec count 0
timer count 120; exec count 0
timer count 121; exec count 0
timer count 122; exec count 0
timer count 123; exec count 0
timer count 124; exec count 0
timer count 125; exec count 0
timer count 126; exec count 0
timer count 127; exec count 0
timer count 128; exec count 0
timer count 129; exec count 0
timer count 130; exec count 0
timer count 131; exec count 0
timer count 132; exec count 0
timer count 133; exec count 0
timer count 134; exec count 0
timer count 135; exec count 0
timer count 136; exec count 0
timer count 137; exec count 0
timer count 138; exec count 0
timer count 139; exec count 0
timer count 140; exec count 0
timer count 141; exec count 0
timer count 142; exec count 0
timer count 143; exec count 0
timer count 144; exec count 0
timer count 145; exec count 0
timer count 146; exec count 0
timer count 147; exec count 0
timer count 148; exec count 0
timer count 149; exec count 0
timer count 150; exec count 0
timer count 151; exec count 0
timer count 152; exec count 0
timer count 153; exec count 0
timer count 154; exec count 0
timer count 155; exec count 0
timer count 156; exec count 0
timer count 157; exec count 0
timer count 158; exec count 0
timer count 159; exec count 0
timer count 160; exec count 0
timer count 161; exec count 0
timer count 162; exec count 0
timer count 163; exec count 0
timer count 164; exec count 0
timer count 165; exec count 0
timer count 166; exec count 0
timer count 167; exec count 0
timer count 168; exec count 0
timer count 169; exec count 0
timer count 170; exec count 0
timer count 171; exec count 0
timer count 172; exec count 0
timer count 173; exec count 0
timer count 174; exec count 0
timer count 175; exec count 0
timer count 176; exec count 0
timer count 177; exec count 0
timer count 178; exec count 0
timer count 179; exec count 0
timer count 180; exec count 0
timer count 181; exec count 0
timer count 182; exec count 0
timer count 183; exec count 0
timer count 184; exec count 0
timer count 185; exec count 0
timer count 186; exec count 0
timer count 187; exec count 0
timer count 188; exec count 0
timer count 189; exec count 0
timer count 190; exec count 0
timer count 191; exec count 0
timer count 192; exec count 0
timer count 193; exec count 0
timer count 194; exec count 0
timer count 195; exec count 0
timer count 196; exec count 0
timer count 197; exec count 0
timer count 198; exec count 0
timer count 199; exec count 0
timer count 200; exec count 0
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 201; exec count 3
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 202; exec count 8
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 203; exec count 13
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 204; exec count 18
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 205; exec count 23
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 206; exec count 27
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 207; exec count 32
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 208; exec count 37
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 209; exec count 42
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 210; exec count 47
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 211; exec count 52
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 212; exec count 56
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 213; exec count 61
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 214; exec count 66
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 215; exec count 71
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 216; exec count 76
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 217; exec count 80
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 218; exec count 85
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 219; exec count 90
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 220; exec count 95
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 221; exec count 100
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 222; exec count 104
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 223; exec count 109
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 224; exec count 114
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 225; exec count 119
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 226; exec count 124
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 227; exec count 129
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 228; exec count 133
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 229; exec count 138
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 230; exec count 143
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 231; exec count 148
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 232; exec count 153
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 233; exec count 157
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 234; exec count 162
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 235; exec count 167
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 236; exec count 172
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 237; exec count 177
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 238; exec count 182
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 239; exec count 186
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 240; exec count 191
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 241; exec count 196
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 242; exec count 201
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 243; exec count 206
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 244; exec count 210
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 245; exec count 215
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 246; exec count 220
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 247; exec count 225
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 248; exec count 229
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 249; exec count 234
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 250; exec count 239
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 251; exec count 244
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
timer count 252; exec count 248
timer task initially on cpu 1
timer task initially on cpu 1
timer task initially on cpu 1
self context switches: 21 vol, 3550 invol
current thread context switches 0 vol, 1 invol
elapsed time 25.263 sec
TEE spin command done
task exec count after spin 252
timer count 253; exec count 252
timer task initially on cpu 1
timer count 254; exec count 253
timer task initially on cpu 1
timer count 255; exec count 254
timer task initially on cpu 1
timer count 256; exec count 255
timer task initially on cpu 1
timer count 257; exec count 256
timer task initially on cpu 1
timer count 258; exec count 257

I thought that perhaps the scheduling tick from the kernel was being treated as an FIQ and causing preemptions, but even after enabling nohz_full on CPU 1, I’m still seeing this behavior, both in QEMU and on a Raspberry Pi.

Therefore, I have the following questions:

  1. Are TAs intended to be preemptible?
  2. If I want to control/limit the amount of preemption that can occur during TA execution (ideally without having to disable all foreign interrupts), how can I do that?
etienne-lms commented 3 weeks ago

Hello @bstandaert-wustl,

Issue https://github.com/OP-TEE/optee_os/issues/4781 is rather about ability to cancel a TA execution, not about whether or not it can be preempted.

  1. Are TAs intended to be preemptible?

TAs are interruptible and preemptible. When a TA executes, it can be interrupted by secure interrupts and by non-secure interrupts.

When a secure interrupt occurs, the TA is interrupted for the secure interrupt to be served then the TA execution resumes.

When a non-secure interrupts occurs, the TA is interrupted and secure monitor switches to non-secure world for the interrupt to be served. When interrupt is served in non-secure world, the non-secure scheduler may decide to schedule another thread. In this case, the TA execution will resume only when the non-secure scheduler decides to schedule back the non-secure thread related to the TA invocation (that is the non-secure thread that initially invoked the TA, e.g. through a call to TEE_InvokeTACommand()).

Non-secure interrupt management and scheduling impacts are described in documentation section Deliver non-secure interrupts to Normal World and section Scheduling considerations in Trusted thread scheduling.

  1. If I want to control/limit the amount of preemption that can occur during TA execution (ideally without having to disable all foreign interrupts), how can I do that?

You need to act on the Linux thread that invokes the TA. OP-TEE core has no impact on non-secure scheduler (aside masking non-secure interrupts when a TA executes).

bstandaert-wustl commented 3 weeks ago

OK, thanks, I think that clarifies some things!