"Detect missing frame + Warning: the value is empty for raft" ERROR when extraction_fps=1

TikaToka commented 1 year ago

Hi, first of all, thank you for sharing your amazing work!

I am trying to extract optical features for Charades-sta dataset with command

python main.py \
    feature_type=raft \
    batch_size=8 \
    extraction_fps=1
    on_extraction=save_numpy \
    device="cuda:4" \
    keep_tmp_files=true \
    video_paths="[/home/jckim/workspace/video_features/tmp/raft/0A8CF_new_fps.mp4]"

(never mind for video_paths the reason for new_fps video's are there is that i've tested if this was problem of re-encoding. neither raw video or re-encoded videos occur same problem) With the command, I could get 1fps re-encoded video. But when extracting features, happens like this

and the features extraceted are all blank.

Is there any way to solve this problem?

Thank you in advance!

v-iashin commented 1 year ago

Hi, could you attach the video file?

TikaToka commented 1 year ago

https://user-images.githubusercontent.com/49054667/218966176-c8d44260-d6e9-4103-87d0-dd6b307b24fe.mp4

sample videos are here

name: WPR8G.mp4 from Charades dataset

v-iashin commented 1 year ago

Sorry for the late response but have you tried to use a smaller batch_size?

I tried to reproduce this problem on google colab but I couldn't. Could you try as well?

https://colab.research.google.com/drive/18I95Rn1B3a2ISfD9b-o4o93m3XuHbcIY?usp=sharing

TikaToka commented 1 year ago

i've also tried with batch_size =1 but same result Also, as i need a npy file and there are a lots of videos in charades, I can't use colab as workspace :(

TikaToka commented 1 year ago

also tried your jupyter file in my server, does not work

TikaToka commented 1 year ago

For extra information) In colab works well as you mentioned.. idk why it does not work only at my server :(

pip freeze result

antlr4-python3-runtime @ file:///home/conda/feedstock_root/build_artifacts/antlr-python-runtime-meta_1638309185939/work asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1670263926556/work attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1659291887007/work backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work certifi==2022.12.7 cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1671179356964/work comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1670575068857/work debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1674522341001/work decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1667317341051/work ffmpeg-python==0.2.0 ftfy @ file:///home/conda/feedstock_root/build_artifacts/ftfy_1647200718722/work future @ file:///home/conda/feedstock_root/build_artifacts/future_1673596611778/work importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1672612343532/work iniconfig @ file:///home/conda/feedstock_root/build_artifacts/iniconfig_1603384189793/work ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1676322140253/work ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1676047456691/work jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1669134318875/work jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1675178684608/work jupyter-core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1675109858760/work llvmlite==0.36.0 matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work mkl-fft==1.2.0 mkl-random==1.1.1 mkl-service==2.3.0 nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1664684991461/work numba @ file:///tmp/build/80754af9/numba_1616774046117/work numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1596233721170/work olefile==0.46 omegaconf @ file:///home/conda/feedstock_root/build_artifacts/omegaconf_1636817774483/work opencv-python==4.4.0.44 packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1637239678211/work parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work Pillow @ file:///tmp/build/80754af9/pillow_1594307295532/work platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1670613602756/work pluggy @ file:///home/conda/feedstock_root/build_artifacts/pluggy_1648772598846/work prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1670414775770/work psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1667885878918/work ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work py @ file:///home/conda/feedstock_root/build_artifacts/py_1636301881863/work pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1672682006896/work pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1652235407899/work pytest==7.1.2 python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work PyYAML==5.3.1 pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1673612649529/work regex @ file:///tmp/build/80754af9/regex_1648447708112/work resampy==0.2.2 scipy @ file:///tmp/build/80754af9/scipy_1597686649129/work six==1.15.0 SoundFile @ file:///home/conda/feedstock_root/build_artifacts/pysoundfile_1607011569782/work stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1644342247877/work torch==1.7.1 torchaudio==0.7.0a0+a853dff torchvision==0.8.2 tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1666788592778/work tqdm @ file:///tmp/build/80754af9/tqdm_1600709023549/work traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1675110562325/work typing-extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1638334978229/work wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1675982654259/work

v-iashin commented 1 year ago

does it work with the provided conda environment?

If not, one option could be to use a docker environment (or a singularity container if docker is not allowed)

What might be the problem? I think you need to check which ffmpeg you are using and try the command:

ffmpeg -i original.mp4 -filter:v fps=fps=1 original_1fps.mp4

and check the video.

TikaToka commented 1 year ago

Also Tried at docker env before.. but same error occurs...)

jckim@vtt-server:~/workspace/video_features$ docker run --gpus '"device=0"' --name=jckim --shm-size 8G -i -t -v /data:/data iashin/video_features:latest /bin/bash

(torch_zoo) ubuntu@8807696ec356:~/video_features$ ffmpeg -i /data/mjjung/Charades_v1/WPR8G.mp4 -filter:v fps=fps=1 WPR8G_1fps.mp4 ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers built with gcc 7.5.0 (crosstool-NG 1.24.0.131_87df0e6_dirty) configuration: --prefix=/home/ubuntu/miniconda3/envs/torch_zoo --cc=/home/conda/feedstock_root/build_artifacts/ffmpeg_1596712246804/_build_env/bin/x86_64-conda-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-gpl --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-libx264 --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame libavutil 56. 51.100 / 56. 51.100 libavcodec 58. 91.100 / 58. 91.100 libavformat 58. 45.100 / 58. 45.100 libavdevice 58. 10.100 / 58. 10.100 libavfilter 7. 85.100 / 7. 85.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 7.100 / 5. 7.100 libswresample 3. 7.100 / 3. 7.100 libpostproc 55. 7.100 / 55. 7.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/data/mjjung/Charades_v1/WPR8G.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf56.40.101 Duration: 00:00:31.30, start: 0.000000, bitrate: 1561 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 1426 kb/s, 16.63 fps, 16.63 tbr, 16631 tbn, 33.26 tbc (default) Metadata: handler_name : VideoHandler Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 131 kb/s (default) Metadata: handler_name : SoundHandler Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264)) Stream #0:1 -> #0:1 (aac (native) -> aac (native)) Press [q] to stop, [?] for help [libx264 @ 0x55c6905c4980] using SAR=1/1 [libx264 @ 0x55c6905c4980] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512 [libx264 @ 0x55c6905c4980] profile High, level 3.1 [libx264 @ 0x55c6905c4980] 264 - core 152 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=22 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=1 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00 Output #0, mp4, to 'WPR8G_1fps.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf58.45.100 Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 1 fps, 16384 tbn, 1 tbc (default) Metadata: handler_name : VideoHandler encoder : Lavc58.91.100 libx264 Side data: cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default) Metadata: handler_name : SoundHandler encoder : Lavc58.91.100 aac frame= 22 fps=0.0 q=0.0 size= 0kB time=00:00:23.35 bitrate= 0.0kframe= 31 fps= 20 q=-1.0 Lsize= 2135kB time=00:00:31.27 bitrate= 559.1kbits/s speed=20.4x
video:1635kB audio:493kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.330588% [libx264 @ 0x55c6905c4980] frame I:2 Avg QP:11.66 size: 67910 [libx264 @ 0x55c6905c4980] frame P:18 Avg QP:13.28 size: 56428 [libx264 @ 0x55c6905c4980] frame B:11 Avg QP:13.64 size: 47428 [libx264 @ 0x55c6905c4980] consecutive B-frames: 45.2% 19.4% 9.7% 25.8% [libx264 @ 0x55c6905c4980] mb I I16..4: 17.6% 57.2% 25.2% [libx264 @ 0x55c6905c4980] mb P I16..4: 14.1% 56.7% 16.5% P16..4: 5.5% 5.4% 1.5% 0.0% 0.0% skip: 0.4% [libx264 @ 0x55c6905c4980] mb B I16..4: 7.2% 26.9% 8.7% B16..8: 20.0% 21.7% 4.9% direct: 9.6% skip: 1.0% L0:50.2% L1:38.5% BI:11.3% [libx264 @ 0x55c6905c4980] 8x8 transform intra:63.8% inter:58.9% [libx264 @ 0x55c6905c4980] coded y,uvDC,uvAC intra: 71.2% 89.1% 68.8% inter: 64.7% 93.9% 26.3% [libx264 @ 0x55c6905c4980] i16 v,h,dc,p: 24% 26% 14% 36% [libx264 @ 0x55c6905c4980] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 28% 26% 13% 4% 4% 5% 8% 5% 6% [libx264 @ 0x55c6905c4980] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 36% 27% 9% 3% 6% 5% 8% 4% 3% [libx264 @ 0x55c6905c4980] i8c dc,h,v,p: 40% 22% 27% 12% [libx264 @ 0x55c6905c4980] Weighted P-Frames: Y:0.0% UV:0.0% [libx264 @ 0x55c6905c4980] ref P L0: 55.8% 16.9% 16.5% 10.7% [libx264 @ 0x55c6905c4980] ref B L0: 83.6% 15.7% 0.8% [libx264 @ 0x55c6905c4980] ref B L1: 91.5% 8.5% [libx264 @ 0x55c6905c4980] kb/s:431.80

Result Video

https://user-images.githubusercontent.com/49054667/219874498-2e441376-923e-4372-bd5b-31a24df96898.mp4

(torch_zoo) ubuntu@8807696ec356:~/video_features$ bash run.sh feature_type: raft extraction_fps: 1 extraction_total: null side_size: null resize_to_smaller_edge: true finetuned_on: sintel device: cuda:0 batch_size: 1 on_extraction: save_numpy output_path: ./output/raft tmp_path: ./tmp/raft keep_tmp_files: true show_pred: false config: null video_paths:

/data/mjjung/Charades_v1/WPR8G.mp4 file_with_video_paths: null

Saving features to ./output/raft Device: cuda:0 The number of specified videos: 1 0%| | 0/1 [00:00<?, ?it/s]Detect missing frame Warning: the value is empty for raft @ ./output/raft/WPR8G_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/WPR8G_timestamps_ms.npy 100%|███████████████████████████████████████| 1/1 [00:01<00:00, 1.71s/it]

TikaToka commented 1 year ago

new information found)

I tried to extract features for all videos, most of the videos had a same problem, but few returned features well.. (about 1~2%? videos succeeded. tried at docker env provided)

(torch_zoo) ubuntu@c4dd7bf2d508:~/video_features$ python main.py \
>     feature_type=raft \
>     on_extraction=save_numpy \
>     keep_tmp_files=true \
>     file_with_video_paths=charades.txt \
>     extraction_fps=1

feature_type: raft extraction_fps: 1 extraction_total: null side_size: null resize_to_smaller_edge: true finetuned_on: sintel device: cuda:0 batch_size: 1 on_extraction: save_numpy output_path: ./output/raft tmp_path: ./tmp/raft keep_tmp_files: true show_pred: false config: null video_paths: null file_with_video_paths: charades.txt

Saving features to ./output/raft Device: cuda:0 The number of specified videos: 9848 0%| | 0/9848 [00:00<?, ?it/s]Detect missing frame Warning: the value is empty for raft @ ./output/raft/JRSPH_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/JRSPH_timestamps_ms.npy 0%| | 1/9848 [00:02<5:49:03, 2.13s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/KVWRK_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/KVWRK_timestamps_ms.npy 0%| | 2/9848 [00:03<5:02:10, 1.84s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/TVBLW_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/TVBLW_timestamps_ms.npy 0%| | 4/9848 [00:16<13:03:53, 4.78s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/X42YA_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/X42YA_timestamps_ms.npy 0%| | 5/9848 [00:18<10:14:36, 3.75s/it]Detect missing frame 0%| | 6/9848 [00:24<12:24:33, 4.54s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/2L6E2_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/2L6E2_timestamps_ms.npy 0%| | 7/9848 [00:27<11:15:52, 4.12s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/D260T_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/D260T_timestamps_ms.npy 0%| | 8/9848 [00:29<9:08:19, 3.34s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/55PFS_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/55PFS_timestamps_ms.npy 0%| | 9/9848 [00:30<7:20:06, 2.68s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/VVMYX_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/VVMYX_timestamps_ms.npy 0%| | 10/9848 [00:31<6:24:43, 2.35s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/5C3NY_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/5C3NY_timestamps_ms.npy 0%| | 12/9848 [00:40<10:02:10, 3.67s/it]Detect missing frame 0%| | 13/9848 [00:46<11:42:47, 4.29s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/9SP9P_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/9SP9P_timestamps_ms.npy 0%| | 14/9848 [00:48<9:32:13, 3.49s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/7YV59_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/7YV59_timestamps_ms.npy

v-iashin / video_features

"Detect missing frame + Warning: the value is empty for raft" ERROR when extraction_fps=1 #91