PicklingError: Can't pickle <function Embedding.forward at 0x7f39a180d750>: it's not the same object as torch.nn.modules.sparse.Embedding.forward

[alpacaduby]

As I run: bash scripts/realtime_cli.sh, I encounter this error:

Traceback (most recent call last):
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/media/Disk1/xxx/code/video/Flash-VStream-main/flash_vstream/serve/cli_video_stream2.py", line 353, in <module>
    main(args)
  File "/media/Disk1/xxx/code/video/Flash-VStream-main/flash_vstream/serve/cli_video_stream2.py", line 258, in main
    p3.start()
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function Embedding.forward at 0x7f39a180d750>: it's not the same object as torch.nn.modules.sparse.Embedding.forward
[2024-07-07 16:39:32,941] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/multiprocessing/synchronize.py", line 110, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory

If I change parser.add_argument("--device", type=str, default="cuda") to cuda:0, this error disappears but I will encounter this one:

Current Time: 16:46:38 Run for: 5.004142999649048
USER: what is in the video?
MemManager Exception: 'NoneType' object has no attribute 'preprocess'
MemManager Exception: 'NoneType' object has no attribute 'preprocess'
Traceback (most recent call last):
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/media/Disk1/Xxx/code/video/Flash-VStream-main/flash_vstream/serve/cli_video_stream2.py", line 353, in <module>
    main(args)
  File "/media/Disk1/Xxx/code/video/Flash-VStream-main/flash_vstream/serve/cli_video_stream2.py", line 302, in main
    output_ids = model.generate(
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/site-packages/transformers/generation/utils.py", line 1538, in generate
    return self.greedy_search(
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/site-packages/transformers/generation/utils.py", line 2362, in greedy_search
    outputs = self(
  File "/home/Xxx/anaconda3/envs/vstream2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/Disk1/Xxx/code/video/Flash-VStream-main/flash_vstream/model/language_model/vstream_llama.py", line 79, in forward
    ) = self.prepare_inputs_labels_for_multimodal_streaming(
  File "/media/Disk1/Xxx/code/video/Flash-VStream-main/flash_vstream/model/vstream_arch.py", line 494, in prepare_inputs_labels_for_multimodal_streaming
    image_features = self.cat_proj(image_features)
  File "/media/Disk1/Xxx/code/video/Flash-VStream-main/flash_vstream/model/vstream_arch.py", line 281, in cat_proj
    feature_embed = torch.cat(all_features, dim=0)
RuntimeError: torch.cat(): expected a non-empty list of Tensors

Could you please tell me how to solve these problems?

[alpacaduby]

Solve the second error by removing if 'vstream' in model_name.lower(): in flash_vstream/model/builder.py

[InvincibleWyq]

Thanks for your comment! We will double check the model_name variable in the code. Also, please refer to the latest repo at https://github.com/IVGSZ/Flash-VStream We will keep update that repo, we are unable to update this original repo due to the end of internship.