search

z-x-yang / Segment-and-Track-Anything

An open-source project dedicated to tracking and segmenting any objects in videos, either automatically or interactively. The primary algorithms utilized include the Segment Anything Model (SAM) for key-frame segmentation and Associating Objects with Transformers (AOT) for efficient tracking and propagation purposes.
GNU Affero General Public License v3.0
2.84k stars 341 forks source link

WebUI cannot display tracking results #49

Closed Manticore425 closed 1 year ago

Manticore425 commented 1 year ago

When I import a video to WebUI, WebUI cannot display the tracking results. At first, the program will report an error:

SegTracker has been initialized
Click
Click
Start tracking !
processed frame 49, obj_num 2
finished
frame 49 writed
/home/weedy/Projects/Segment-and-Track-Anything/tracking_results/blackswan/blackswan_seg.mp4 saved

finished
Traceback (most recent call last):
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/blocks.py", line 1323, in process_api
    result = await self.call_function(
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/blocks.py", line 1051, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/weedy/Projects/Segment-and-Track-Anything/app.py", line 251, in tracking_objects
    return tracking_objects_in_video(Seg_Tracker, input_video, input_img_seq, fps)
  File "/home/weedy/Projects/Segment-and-Track-Anything/seg_track_anything.py", line 94, in tracking_objects_in_video
    return video_type_input_tracking(SegTracker, input_video, io_args, video_name)
  File "/home/weedy/Projects/Segment-and-Track-Anything/seg_track_anything.py", line 196, in video_type_input_tracking
    imageio.mimsave(io_args['output_gif'], masked_pred_list, fps=fps)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/imageio/v2.py", line 484, in mimwrite
    return file.write(ims, is_batch=True, **kwargs)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/imageio/plugins/pillow.py", line 354, in write
    raise TypeError(
TypeError: The keyword `fps` is no longer supported. Use `duration`(in ms) instead, e.g. `fps=50` == `duration=20` (1000 * 1/50).

So I changed the 196th line of the file "seg_track_anything. py": imageio. mimsave (io_args ['output_gif '], masked_pred_list, fps=fps) to imageio. mimsave (io_args [" output_gif "], masked_pred_list, duration=int (1000/fps))

But WebUI still cannot output results. It will report an error:

Start tracking !
processed frame 49, obj_num 1
finished
frame 49 writed
/home/weedy/Projects/Segment-and-Track-Anything/tracking_results/blackswan/blackswan_seg.mp4 saved

finished
/home/weedy/Projects/Segment-and-Track-Anything/tracking_results/blackswan/blackswan_seg.gif saved
sh: 1: zip: not found
/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py:2239: UserWarning: Video does not have browser-compatible container or codec. Converting to mp4
  warnings.warn(
Traceback (most recent call last):
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/blocks.py", line 1326, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/blocks.py", line 1260, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py", line 2824, in postprocess
    "name": self.make_temp_copy_if_needed(y),
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py", line 259, in make_temp_copy_if_needed
    temp_dir = self.hash_file(file_path)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py", line 223, in hash_file
    with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/weedy/Projects/Segment-and-Track-Anything/tracking_results/blackswan/blackswan_pred_mask.zip'

How should I fix that bug?

gptbert commented 1 year ago

pip install imageio==2.5.0

Manticore425 commented 1 year ago

pip install imageio==2.5.0

Thank you, but that doesn't help. I have changed back to: imageio. mimsave (io_args ['output_gif '], masked_pred_list, fps=fps)

It reports:

SegTracker has been initialized
Click
Start tracking !
processed frame 49, obj_num 2
finished
frame 49 writed
/home/weedy/Projects/Segment-and-Track-Anything/tracking_results/blackswan/blackswan_seg.mp4 saved

finished
/home/weedy/Projects/Segment-and-Track-Anything/tracking_results/blackswan/blackswan_seg.gif saved
sh: 1: zip: not found
/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py:2239: UserWarning: Video does not have browser-compatible container or codec. Converting to mp4
  warnings.warn(
Traceback (most recent call last):
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/blocks.py", line 1326, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/blocks.py", line 1260, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py", line 2824, in postprocess
    "name": self.make_temp_copy_if_needed(y),
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py", line 259, in make_temp_copy_if_needed
    temp_dir = self.hash_file(file_path)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py", line 223, in hash_file
    with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/weedy/Projects/Segment-and-Track-Anything/tracking_results/blackswan/blackswan_pred_mask.zip'
gptbert commented 1 year ago

pip install imageio==2.5.0

Thank you, but that doesn't help. I have changed back to: imageio. mimsave (io_args ['output_gif '], masked_pred_list, fps=fps)

It reports:

SegTracker has been initialized
Click
Start tracking !
processed frame 49, obj_num 2
finished
frame 49 writed
/home/weedy/Projects/Segment-and-Track-Anything/tracking_results/blackswan/blackswan_seg.mp4 saved

finished
/home/weedy/Projects/Segment-and-Track-Anything/tracking_results/blackswan/blackswan_seg.gif saved
sh: 1: zip: not found
/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py:2239: UserWarning: Video does not have browser-compatible container or codec. Converting to mp4
  warnings.warn(
Traceback (most recent call last):
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/blocks.py", line 1326, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/blocks.py", line 1260, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py", line 2824, in postprocess
    "name": self.make_temp_copy_if_needed(y),
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py", line 259, in make_temp_copy_if_needed
    temp_dir = self.hash_file(file_path)
  File "/home/weedy/miniconda3/envs/samtrack/lib/python3.10/site-packages/gradio/components.py", line 223, in hash_file
    with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/weedy/Projects/Segment-and-Track-Anything/tracking_results/blackswan/blackswan_pred_mask.zip'
sh: 1: zip: not found

please install zip at first

gptbert commented 1 year ago

Update seg_track_anything.py The mp4 files generated by imageio.mimsave() can be played in gradio video

https://github.com/gptbert/Segment-and-Track-Anything/blob/main/seg_track_anything.py

    ##################
    # Visualization
    ##################

    # draw pred mask on frame and save as a video
    cap = cv2.VideoCapture(input_video)
    fps = cap.get(cv2.CAP_PROP_FPS)
    # width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    # height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    # num_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

    # fourcc = cv2.VideoWriter_fourcc(*"mp4v")
    # if input_video[-3:]=='mp4':
    #     fourcc =  cv2.VideoWriter_fourcc(*"mp4v")
    # elif input_video[-3:] == 'avi':
    #     fourcc =  cv2.VideoWriter_fourcc(*"MJPG")
    #     # fourcc = cv2.VideoWriter_fourcc(*"XVID")
    # else:
    #     fourcc = int(cap.get(cv2.CAP_PROP_FOURCC))
    # out = cv2.VideoWriter(io_args["output_video"], fourcc, fps, (width, height))

    frame_idx = 0
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break

        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        pred_mask = pred_list[frame_idx]
        masked_frame = draw_mask(frame, pred_mask)
        cv2.imwrite(
            f"{io_args['output_masked_frame_dir']}/{str(frame_idx).zfill(5)}.png",
            masked_frame[:, :, ::-1],
        )

        masked_pred_list.append(masked_frame)
        # masked_frame = cv2.cvtColor(masked_frame, cv2.COLOR_RGB2BGR)
        # out.write(masked_frame)
        print(f"frame {frame_idx} writed", end="\r")
        frame_idx += 1
    # out.release()
    cap.release()

    imageio.mimsave(io_args["output_video"], masked_pred_list, fps=fps)
    print("\n{} saved".format(io_args["output_video"]))
    print("\nfinished")
Manticore425 commented 1 year ago

Update seg_track_anything.py The mp4 files generated by imageio.mimsave() can be played in gradio video

https://github.com/gptbert/Segment-and-Track-Anything/blob/main/seg_track_anything.py

    ##################
    # Visualization
    ##################

    # draw pred mask on frame and save as a video
    cap = cv2.VideoCapture(input_video)
    fps = cap.get(cv2.CAP_PROP_FPS)
    # width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    # height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    # num_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

    # fourcc = cv2.VideoWriter_fourcc(*"mp4v")
    # if input_video[-3:]=='mp4':
    #     fourcc =  cv2.VideoWriter_fourcc(*"mp4v")
    # elif input_video[-3:] == 'avi':
    #     fourcc =  cv2.VideoWriter_fourcc(*"MJPG")
    #     # fourcc = cv2.VideoWriter_fourcc(*"XVID")
    # else:
    #     fourcc = int(cap.get(cv2.CAP_PROP_FOURCC))
    # out = cv2.VideoWriter(io_args["output_video"], fourcc, fps, (width, height))

    frame_idx = 0
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break

        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        pred_mask = pred_list[frame_idx]
        masked_frame = draw_mask(frame, pred_mask)
        cv2.imwrite(
            f"{io_args['output_masked_frame_dir']}/{str(frame_idx).zfill(5)}.png",
            masked_frame[:, :, ::-1],
        )

        masked_pred_list.append(masked_frame)
        # masked_frame = cv2.cvtColor(masked_frame, cv2.COLOR_RGB2BGR)
        # out.write(masked_frame)
        print(f"frame {frame_idx} writed", end="\r")
        frame_idx += 1
    # out.release()
    cap.release()

    imageio.mimsave(io_args["output_video"], masked_pred_list, fps=fps)
    print("\n{} saved".format(io_args["output_video"]))
    print("\nfinished")

Thank you very much! The problem has been resolved.