Video streaming=True output displays nothing

[zjysteven]

Describe the bug

Hi, thanks for this great work. I was trying to use the new video real-time streaming output feature that is enabled by #8906 (currently under the 5.0-dev branch).

I first ran the example demo/stream_video_out/run.py (this) and it worked like a charm. However when I adapted it to my application, the video block displayed nothing despite reporting no errors (there was a video progress bar shown but with timestamp of 00:00 / Nan:Nan).

Interestingly, I confirmed that there were video segments saved and they could be played normally with my local video player. Furthermore, after all video segments were generated, if I run another gradio app that reads and plays the video segments with gr.Video(streaming=True), then it could work smoothly.

I completely got lost with this weird "bug" and would appreciate any thoughts/ideas/guesses on what I was doing wrong. @abidlabs @freddyaboulton

Have you searched existing issues? 🔎

• [X] I have searched and found no existing issues

Reproduction

Unfortunately I don't have a minimal reproducible example because it is an app that involves custom diffusion model and complicated environment setup. But the essence of the code is below.

# this is content of the main function whose return value is fed to gr.Video(streaming=True)
def main(video_url, video_file, prompt):
    ...

    n_chunks = 0
    name = os.path.join(VIDEO_DIR, f"output_{n_chunks}{'.mp4' if stream_as_mp4 else '.ts'}")
    segment_file = cv2.VideoWriter(name, video_codec, fps, (width, height)) # type: ignore

    for i in range(batches):
        output_frames = ... # process each batch of frames

        for j in range(len(output_frames)):
            video_result[frame_cnt] = postprocess_image(
                output_frames[j].permute(1, 2, 0),
                output_type="pt"
            )
            segment_file.write(
                cv2.cvtColor(
                    np.array(TF.to_pil_image(video_result[frame_cnt].permute(2, 0, 1))),
                    cv2.COLOR_BGR2RGB
                )
            )
            frame_cnt += 1

        n_chunks += 1
        segment_file.release()
        yield name
        name = os.path.join(VIDEO_DIR, f"output_{n_chunks}{'.mp4' if stream_as_mp4 else '.ts'}")
        segment_file = cv2.VideoWriter(name, video_codec, fps, (width, height)) # type: ignore

# this is the definition of the components
with gr.Blocks() as demo:
    real_time_output = gr.Video(
        label="Real-Time Decoding Output", 
        streaming=True, 
        autoplay=True, 
        elem_id="stream_video_output"
    )

    submit_btn = gr.Button("Run", scale=1)

    submit_btn.click(
        fn=main, 
        inputs=[video_url, video_file, prompt_input], 
        outputs=[real_time_output]
    )

Screenshot

Please see this video recording where the streamed video output should be displayed in "Real-Time Decoding Output" but it just showed empty. Screencast from 08-16-2024 08_21_42 PM.webm

Logs

No response

System Info

Gradio Environment Information:
------------------------------
Operating System: Linux
gradio version: 4.41.0
gradio_client version: 1.3.0

------------------------------------------------
gradio dependencies in your environment:

aiofiles: 23.2.1
anyio: 4.4.0
fastapi: 0.111.0
ffmpy: 0.3.2
gradio-client==1.3.0 is not installed.
httpx: 0.27.0
huggingface-hub: 0.22.2
importlib-resources: 6.4.0
jinja2: 3.1.3
markupsafe: 2.1.5
matplotlib: 3.9.1
numpy: 1.26.4
orjson: 3.10.6
packaging: 24.0
pandas: 2.2.2
pillow: 10.3.0
pydantic: 2.8.2
pydub: 0.25.1
python-multipart: 0.0.9
pyyaml: 6.0.1
ruff: 0.5.1
semantic-version: 2.10.0
tomlkit==0.12.0 is not installed.
typer: 0.12.3
typing-extensions: 4.11.0
urllib3: 2.2.2
uvicorn: 0.30.1
authlib; extra == 'oauth' is not installed.
itsdangerous; extra == 'oauth' is not installed.

gradio_client dependencies in your environment:

fsspec: 2024.2.0
httpx: 0.27.0
huggingface-hub: 0.22.2
packaging: 24.0
typing-extensions: 4.11.0
websockets: 11.0.3

Severity

I can work around it

[freddyaboulton]

What video_codec are you using @zjysteven ? If you could share a small realistic-ish repro then we could debug this better!

[zjysteven]

Thanks for the reply. I have tried several codecs but had no luck in getting the video displayed. It's quite difficult for me to come up with a repro script. I guess I will close for now and reopen if I somehow have it ready.

[freddyaboulton]

Ok @zjysteven - i'll still be looking into it so reach out with any questions!

[zjysteven]

Hi @freddyaboulton, I finally identified the thing. So if one creates a public URL with demo.launch(share=True), the video will not be displayed no matter if the demo is accessed via local or public URL. demo.lauch(share=False) works fine. This can be confirmed with the official example.

[freddyaboulton]

Thanks for isolating the issue @zjysteven !

[zjysteven]

Happy to help identify the cause. Kindly keep me posted when there's a fix of this. We are very in need of this streaming feature for our demo.