Open TikaToka opened 1 year ago
Hi, could you attach the video file?
sample videos are here
name: WPR8G.mp4 from Charades dataset
Sorry for the late response but have you tried to use a smaller batch_size
?
I tried to reproduce this problem on google colab but I couldn't. Could you try as well?
https://colab.research.google.com/drive/18I95Rn1B3a2ISfD9b-o4o93m3XuHbcIY?usp=sharing
i've also tried with batch_size =1 but same result
Also, as i need a npy file and there are a lots of videos in charades, I can't use colab as workspace :(
also tried your jupyter file in my server, does not work
For extra information) In colab works well as you mentioned.. idk why it does not work only at my server :(
antlr4-python3-runtime @ file:///home/conda/feedstock_root/build_artifacts/antlr-python-runtime-meta_1638309185939/work asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1670263926556/work attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1659291887007/work backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work certifi==2022.12.7 cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1671179356964/work comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1670575068857/work debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1674522341001/work decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1667317341051/work ffmpeg-python==0.2.0 ftfy @ file:///home/conda/feedstock_root/build_artifacts/ftfy_1647200718722/work future @ file:///home/conda/feedstock_root/build_artifacts/future_1673596611778/work importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1672612343532/work iniconfig @ file:///home/conda/feedstock_root/build_artifacts/iniconfig_1603384189793/work ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1676322140253/work ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1676047456691/work jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1669134318875/work jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1675178684608/work jupyter-core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1675109858760/work llvmlite==0.36.0 matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work mkl-fft==1.2.0 mkl-random==1.1.1 mkl-service==2.3.0 nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1664684991461/work numba @ file:///tmp/build/80754af9/numba_1616774046117/work numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1596233721170/work olefile==0.46 omegaconf @ file:///home/conda/feedstock_root/build_artifacts/omegaconf_1636817774483/work opencv-python==4.4.0.44 packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1637239678211/work parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work Pillow @ file:///tmp/build/80754af9/pillow_1594307295532/work platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1670613602756/work pluggy @ file:///home/conda/feedstock_root/build_artifacts/pluggy_1648772598846/work prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1670414775770/work psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1667885878918/work ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work py @ file:///home/conda/feedstock_root/build_artifacts/py_1636301881863/work pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1672682006896/work pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1652235407899/work pytest==7.1.2 python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work PyYAML==5.3.1 pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1673612649529/work regex @ file:///tmp/build/80754af9/regex_1648447708112/work resampy==0.2.2 scipy @ file:///tmp/build/80754af9/scipy_1597686649129/work six==1.15.0 SoundFile @ file:///home/conda/feedstock_root/build_artifacts/pysoundfile_1607011569782/work stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1644342247877/work torch==1.7.1 torchaudio==0.7.0a0+a853dff torchvision==0.8.2 tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1666788592778/work tqdm @ file:///tmp/build/80754af9/tqdm_1600709023549/work traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1675110562325/work typing-extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1638334978229/work wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1675982654259/work
does it work with the provided conda environment?
If not, one option could be to use a docker environment (or a singularity container if docker is not allowed)
What might be the problem? I think you need to check which ffmpeg
you are using and try the command:
ffmpeg -i original.mp4 -filter:v fps=fps=1 original_1fps.mp4
and check the video.
Also Tried at docker env before.. but same error occurs...)
jckim@vtt-server:~/workspace/video_features$ docker run --gpus '"device=0"' --name=jckim --shm-size 8G -i -t -v /data:/data iashin/video_features:latest /bin/bash
(torch_zoo) ubuntu@8807696ec356:~/video_features$ ffmpeg -i /data/mjjung/Charades_v1/WPR8G.mp4 -filter:v fps=fps=1 WPR8G_1fps.mp4
ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 7.5.0 (crosstool-NG 1.24.0.131_87df0e6_dirty)
configuration: --prefix=/home/ubuntu/miniconda3/envs/torch_zoo --cc=/home/conda/feedstock_root/build_artifacts/ffmpeg_1596712246804/_build_env/bin/x86_64-conda-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-gpl --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-libx264 --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame
libavutil 56. 51.100 / 56. 51.100
libavcodec 58. 91.100 / 58. 91.100
libavformat 58. 45.100 / 58. 45.100
libavdevice 58. 10.100 / 58. 10.100
libavfilter 7. 85.100 / 7. 85.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 7.100 / 5. 7.100
libswresample 3. 7.100 / 3. 7.100
libpostproc 55. 7.100 / 55. 7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/data/mjjung/Charades_v1/WPR8G.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.40.101
Duration: 00:00:31.30, start: 0.000000, bitrate: 1561 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 1426 kb/s, 16.63 fps, 16.63 tbr, 16631 tbn, 33.26 tbc (default)
Metadata:
handler_name : VideoHandler
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 131 kb/s (default)
Metadata:
handler_name : SoundHandler
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[libx264 @ 0x55c6905c4980] using SAR=1/1
[libx264 @ 0x55c6905c4980] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[libx264 @ 0x55c6905c4980] profile High, level 3.1
[libx264 @ 0x55c6905c4980] 264 - core 152 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=22 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=1 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'WPR8G_1fps.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf58.45.100
Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 1 fps, 16384 tbn, 1 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc58.91.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
Metadata:
handler_name : SoundHandler
encoder : Lavc58.91.100 aac
frame= 22 fps=0.0 q=0.0 size= 0kB time=00:00:23.35 bitrate= 0.0kframe= 31 fps= 20 q=-1.0 Lsize= 2135kB time=00:00:31.27 bitrate= 559.1kbits/s speed=20.4x
video:1635kB audio:493kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.330588%
[libx264 @ 0x55c6905c4980] frame I:2 Avg QP:11.66 size: 67910
[libx264 @ 0x55c6905c4980] frame P:18 Avg QP:13.28 size: 56428
[libx264 @ 0x55c6905c4980] frame B:11 Avg QP:13.64 size: 47428
[libx264 @ 0x55c6905c4980] consecutive B-frames: 45.2% 19.4% 9.7% 25.8%
[libx264 @ 0x55c6905c4980] mb I I16..4: 17.6% 57.2% 25.2%
[libx264 @ 0x55c6905c4980] mb P I16..4: 14.1% 56.7% 16.5% P16..4: 5.5% 5.4% 1.5% 0.0% 0.0% skip: 0.4%
[libx264 @ 0x55c6905c4980] mb B I16..4: 7.2% 26.9% 8.7% B16..8: 20.0% 21.7% 4.9% direct: 9.6% skip: 1.0% L0:50.2% L1:38.5% BI:11.3%
[libx264 @ 0x55c6905c4980] 8x8 transform intra:63.8% inter:58.9%
[libx264 @ 0x55c6905c4980] coded y,uvDC,uvAC intra: 71.2% 89.1% 68.8% inter: 64.7% 93.9% 26.3%
[libx264 @ 0x55c6905c4980] i16 v,h,dc,p: 24% 26% 14% 36%
[libx264 @ 0x55c6905c4980] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 28% 26% 13% 4% 4% 5% 8% 5% 6%
[libx264 @ 0x55c6905c4980] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 36% 27% 9% 3% 6% 5% 8% 4% 3%
[libx264 @ 0x55c6905c4980] i8c dc,h,v,p: 40% 22% 27% 12%
[libx264 @ 0x55c6905c4980] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x55c6905c4980] ref P L0: 55.8% 16.9% 16.5% 10.7%
[libx264 @ 0x55c6905c4980] ref B L0: 83.6% 15.7% 0.8%
[libx264 @ 0x55c6905c4980] ref B L1: 91.5% 8.5%
[libx264 @ 0x55c6905c4980] kb/s:431.80
Result Video
(torch_zoo) ubuntu@8807696ec356:~/video_features$ bash run.sh
feature_type: raft
extraction_fps: 1
extraction_total: null
side_size: null
resize_to_smaller_edge: true
finetuned_on: sintel
device: cuda:0
batch_size: 1
on_extraction: save_numpy
output_path: ./output/raft
tmp_path: ./tmp/raft
keep_tmp_files: true
show_pred: false
config: null
video_paths:
Saving features to ./output/raft Device: cuda:0 The number of specified videos: 1 0%| | 0/1 [00:00<?, ?it/s]Detect missing frame Warning: the value is empty for raft @ ./output/raft/WPR8G_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/WPR8G_timestamps_ms.npy 100%|███████████████████████████████████████| 1/1 [00:01<00:00, 1.71s/it]
I tried to extract features for all videos, most of the videos had a same problem, but few returned features well.. (about 1~2%? videos succeeded. tried at docker env provided)
(torch_zoo) ubuntu@c4dd7bf2d508:~/video_features$ python main.py \
> feature_type=raft \
> on_extraction=save_numpy \
> keep_tmp_files=true \
> file_with_video_paths=charades.txt \
> extraction_fps=1
feature_type: raft extraction_fps: 1 extraction_total: null side_size: null resize_to_smaller_edge: true finetuned_on: sintel device: cuda:0 batch_size: 1 on_extraction: save_numpy output_path: ./output/raft tmp_path: ./tmp/raft keep_tmp_files: true show_pred: false config: null video_paths: null file_with_video_paths: charades.txt
Saving features to ./output/raft Device: cuda:0 The number of specified videos: 9848 0%| | 0/9848 [00:00<?, ?it/s]Detect missing frame Warning: the value is empty for raft @ ./output/raft/JRSPH_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/JRSPH_timestamps_ms.npy 0%| | 1/9848 [00:02<5:49:03, 2.13s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/KVWRK_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/KVWRK_timestamps_ms.npy 0%| | 2/9848 [00:03<5:02:10, 1.84s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/TVBLW_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/TVBLW_timestamps_ms.npy 0%| | 4/9848 [00:16<13:03:53, 4.78s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/X42YA_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/X42YA_timestamps_ms.npy 0%| | 5/9848 [00:18<10:14:36, 3.75s/it]Detect missing frame 0%| | 6/9848 [00:24<12:24:33, 4.54s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/2L6E2_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/2L6E2_timestamps_ms.npy 0%| | 7/9848 [00:27<11:15:52, 4.12s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/D260T_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/D260T_timestamps_ms.npy 0%| | 8/9848 [00:29<9:08:19, 3.34s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/55PFS_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/55PFS_timestamps_ms.npy 0%| | 9/9848 [00:30<7:20:06, 2.68s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/VVMYX_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/VVMYX_timestamps_ms.npy 0%| | 10/9848 [00:31<6:24:43, 2.35s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/5C3NY_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/5C3NY_timestamps_ms.npy 0%| | 12/9848 [00:40<10:02:10, 3.67s/it]Detect missing frame 0%| | 13/9848 [00:46<11:42:47, 4.29s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/9SP9P_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/9SP9P_timestamps_ms.npy 0%| | 14/9848 [00:48<9:32:13, 3.49s/it]Detect missing frame Warning: the value is empty for raft @ ./output/raft/7YV59_raft.npy Warning: the value is empty for timestamps_ms @ ./output/raft/7YV59_timestamps_ms.npy
Hi, first of all, thank you for sharing your amazing work!
I am trying to extract optical features for Charades-sta dataset with command
(never mind for video_paths the reason for new_fps video's are there is that i've tested if this was problem of re-encoding. neither raw video or re-encoded videos occur same problem) With the command, I could get 1fps re-encoded video. But when extracting features,
happens like this
and the features extraceted are all blank.
![image](https://user-images.githubusercontent.com/49054667/218959310-357eaf4a-5f5a-4d3b-a387-39f2d2310489.png)
Is there any way to solve this problem?
Thank you in advance!