Why preempt reset the GPU need to execute `be_stream_device_queue_cap` times?

[flyflypeng]

https://github.com/SJTU-IPADS/reef/blob/0a25de5d60edaef524752a921a8c72e131137879/src/reef/server/scheduler.cpp#L332

I found the GPUResetCU() function actually call ioctl kfd_ioctl_wave_reset_args function to reset all the weavfronts, so why this place need to reset be_stream_device_queue_cap times?

[francis0407]

This is just a small "optimization" to make sure the running kernels are killed.

Consider a naive implementation of the preemption:

1. reset host queue
2. set preemption flag (using preemption stream)
3. wait for preemption stream
4. reset the CUs (after this step, all the kernel will detect the preemption flag as "true")
5. wait for the execution streams if necessary

Actually, the third step "wait for preemption stream" may take several microseconds. To overlap this operation with other steps, we exchange the third and fourth steps. See: https://github.com/SJTU-IPADS/reef/blob/0a25de5d60edaef524752a921a8c72e131137879/src/reef/server/scheduler.cpp#L335

However, this may lead to another problem: the preemption flag might not have been set to "true" when the running kernel is killed. Therefore, to make sure the all the kernels are either killed or evicted by the preemption flag, we repeat the "ResetCU" call several times here.

[flyflypeng]

Thanks for your detailed explanation! But I still have a question, who checks the preempt_flag of best effort task? the amdgpu driver or HIP runtime?

preempt_flag is passed as one of the kernel parameters, but I can't find the place to process this flag. https://github.com/SJTU-IPADS/reef/blob/0a25de5d60edaef524752a921a8c72e131137879/src/reef/executor/hybrid_executor.cpp#L83

[francis0407]

The preemption flag is checked in the device code (i.e., GPU kernels).

https://github.com/SJTU-IPADS/reef/blob/0a25de5d60edaef524752a921a8c72e131137879/resource/mocked_kernel/mocked_kernel.be.cu#L30

[flyflypeng]

Thanks a lot!