exeldro / obs-virtual-cam-filter

GNU General Public License v2.0
121 stars 13 forks source link

Race condition in virtual_cam_filter_source_destroy() causes AV #5

Closed adr26 closed 3 years ago

adr26 commented 3 years ago

Access violations are seen when removing the Virtual Camera filter.

Calling obs_output_stop(context->virtualCam) will trigger asynchronous execution of the end_data_capture_thread thread:

>   obs.dll!obs_output_end_data_capture_internal(obs_output * output, bool signal) Line 2327    C
    win-dshow.dll!virtualcam_stop(void * data, unsigned __int64 ts) Line 74 C
    obs.dll!obs_output_actual_stop(obs_output * output, bool force, unsigned __int64 ts) Line 394   C
    virtual-cam-filter.dll!virtual_cam_filter_source_destroy(void * data) Line 170  C
    obs.dll!obs_source_destroy(obs_source * source) Line 649    C
    obs.dll!obs_source_release(obs_source * source) Line 733    C
    [Inline Frame] obs64.exe!OBSRef<obs_source *,&obs_source_addref,&obs_source_release>::{dtor}() Line 71  C++
    obs64.exe!OBSBasicFilters::on_removeEffectFilter_clicked() Line 887 C++

The end_data_capture_thread thread will then call stop_raw_video():

    vfbasics.dll!AVrfp_ucrt__aligned_free() Unknown
>   [Inline Frame] obs.dll!video_frame_free(video_frame *) Line 35  C
    obs.dll!video_input_free(video_input * input) Line 56   C
    obs.dll!video_output_disconnect(video_output * video, void(*)(void *, video_data *) callback, void * param) Line 421    C
    obs.dll!end_data_capture_thread(void * data) Line 2273  C
    w32-pthreads.dll!ptw32_threadStart(void * vthreadParms) Line 225    C
    ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1>()  Unknown
    vfbasics.dll!AVrfpStandardThreadFunction()  Unknown
    kernel32.dll!BaseThreadInitThunk()  Unknown
    ntdll.dll!RtlUserThreadStart()  Unknown

This races with video_output_close(context->video_output) and causes heap corruption / access violations:

    ucrtbase.dll!_aligned_free()    Unknown
    vfbasics.dll!AVrfp_ucrt__aligned_free() Unknown
    [Inline Frame] obs.dll!video_frame_free(video_frame *) Line 35  C
>   [Inline Frame] obs.dll!video_input_free(video_input *) Line 56  C
    obs.dll!video_output_close(video_output * video) Line 274   C
    virtual-cam-filter.dll!virtual_cam_filter_source_destroy(void * data) Line 176  C
    obs.dll!obs_source_destroy(obs_source * source) Line 649    C
    obs.dll!obs_source_release(obs_source * source) Line 733    C
    [Inline Frame] obs64.exe!OBSRef<obs_source *,&obs_source_addref,&obs_source_release>::{dtor}() Line 71  C++
    obs64.exe!OBSBasicFilters::on_removeEffectFilter_clicked() Line 887 C++

Adding the (missing) call to obs_output_release(context->virtualCam) will trigger sync against end_data_capture_thread in obs_output_destroy(). Similarly, video_output_stop(context->video_output) should be added too (cf. code in preview_output_stop() in \UI\frontend-plugins\decklink-output-ui\decklink-ui-main.cpp).