PKU-YuanGroup / Video-LLaVA

【EMNLP 2024🔥】Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
https://arxiv.org/pdf/2311.10122.pdf
Apache License 2.0
2.92k stars 212 forks source link

Error while feeding the filter graph #106

Open Chenjm08 opened 8 months ago

Chenjm08 commented 8 months ago

有些视频处理时,会出现以下的问题,要怎么解决呢?

[h264 @ 0xa6dd2240] Invalid NAL unit size (12771 > 6598).
[h264 @ 0xa6dd2240] Error splitting the input into NAL units.

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/fastapi/routing.py", line 274, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "image_video_caption_api.py", line 154, in generate_captions
    outputs = video_llava.video_caption(path=filePath, prompt=prompt, system=system)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/image_video_llava.py", line 27, in video_caption
    return self.caption(processor=self.video_processor, path=path, prompt=prompt, system=system, data_type='video')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/image_video_llava.py", line 38, in caption
    tensor = processor.preprocess(path, return_tensors='pt')['pixel_values']
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/llava/model/multimodal_encoder/languagebind/video/processing_video.py", line 156, in preprocess
    return self.__call__(images=images, return_tensors=return_tensors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/llava/model/multimodal_encoder/languagebind/video/processing_video.py", line 141, in __call__
    image_features = [self.image_processor(image, self.transform,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/llava/model/multimodal_encoder/languagebind/video/processing_video.py", line 141, in <listcomp>
    image_features = [self.image_processor(image, self.transform,
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/llava/model/multimodal_encoder/languagebind/video/processing_video.py", line 94, in load_and_transform_video
    video_data = decord_vr.get_batch(frame_id_list)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/decord/video_reader.py", line 175, in get_batch
    arr = _CAPI_VideoReaderGetBatch(self._handle, indices)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/decord/_ffi/_ctypes/function.py", line 173, in __call__
    check_call(_LIB.DECORDFuncCall(
  File "/home/beauty/Video-LLaVA/venv/lib/python3.11/site-packages/decord/_ffi/base.py", line 78, in check_call
    raise DECORDError(err_str)
decord._ffi.base.DECORDError: [11:20:48] /github/workspace/src/video/ffmpeg/threaded_decoder.cc:292: [11:20:47] /github/workspace/src/video/ffmpeg/filter_graph.cc:100: Check failed: av_buffersrc_add_frame_flags(buffersrc_ctx_, frame, AV_BUFFERSRC_FLAG_KEEP_REF) >= 0 (-22 vs. 0) Error while feeding the filter graph
LinB203 commented 8 months ago

应该不会抛出异常错误,我们有处理一些坏数据的代码,会重新取数据。也许这个数据是坏的,你可以把他剔除。 It should not raise error because we have except code to re-pick a sample. https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/train/train.py#L800