Installation Tutorial安装教程 (Chinese and English)

[Ningoose]

序：我因为这个项目受了不少折磨，也不清楚为什么这个Github项目的issue这么少，所以我能得到的帮助很少。我在成功地运行这个项目后，决定写下这个教程帮助后来人。
Preface: I was troubled a lot because of this project, and I don't know why there are so few issues in this Github project, and I get very little help. After successfully running this project, I decided to write this tutorial to help later people.

一、请使用Ubuntu或者MacOS或其他Linux系统！不要使用Windows安装这个项目！相信我，不然你绝对会很痛苦的。我的系统是Ubuntu2204，CUDA 11.0，CuDNN 8.0。 (Please use Ubuntu or MacOS or other Linux systems! Do not use Windows to install this project! Believe me, otherwise you will definitely suffer.My system is Ubuntu2204,CUDA 11.0,CuDNN 8.0)

二、首先一个大问题，作者给的requirement.txt有错误。请按照我给的内容，复制粘贴到requirement.txt里。 (First,a big problem is that the requirement.txt given by the author has errors. Please copy and paste the content I gave into requirements.txt：)

numpy==1.21.6
torch==1.4.0
torchvision==0.5.0
tqdm==4.28.1
yacs==0.1.6
h5py==2.10.0
scipy==1.4.1
numba==0.49.1
smplx==0.1.28
gdown==3.6.4
PyYAML==5.4
joblib==0.14.1
pillow==8.1.2
trimesh==3.5.25
pyrender==0.1.36
progress==1.5
pyglet==1.5.9
filterpy==1.4.5
matplotlib==3.1.3
tensorflow==2.4.1
torchvision==0.5.0
scikit-image==0.16.2
scikit-video==1.1.11
scikit-learn==1.0.2
opencv-python==4.1.2.30
imutils==0.5.4
git+https://github.com/giacaglia/pytube.git
git+https://github.com/mattloper/chumpy.git
git+https://github.com/mkocabas/yolov3-pytorch.git
git+https://github.com/mkocabas/multi-person-tracker.git

然后运行(Then run)：pip install -r requirements.txt

三、boost安装要点： (Boost installation points:) 1.首先使用这个命令（First use this command） sudo apt-get install libboost-dev 2.确保自己本地安装了gcc和g++(Make sure you have gcc and g++ installed locally)

    sudo apt-get install gcc
    sudo apt-get install g++

3.下载mesh项目，cd mesh，确保所在目录下有Cmake文件，输入make all即可(Download the mesh project, cd mesh, make sure there is a Cmake file in the directory, and enter make all) BOOST_INCLUDE_DIRS=/path/to/boost/include make all

四、现在进入到Vid2Doppler中的Python文件夹，并这个命令替换环境中psbody 安装的meshviewer.py： (Now enter the Python folder in Vid2Doppler and replace the meshviewer.py installed by psbody in the environment with this command:) cp meshviewer.py $CONDA_PREFIX/lib/python3.7/site-packages/psbody/mesh/meshviewer.py

五、参照 #1 这里的回答，去vibe下载models整个文件夹，放到lib文件夹内 (Refer to the answer in #1 here, go to vibe to download the entire folder of models and put it in the lib folder)

六、还是在Python文件夹目录下，我建议不要用作者给的prepare_data.sh来下载，直接用里面的链接去浏览器下载即可，下载的是一个压缩包。然后在Vid2doppler目录新建一个data文件夹，里面再新建一个vibe_data文件夹，将压缩包解压到vibe_data里。同样的，分类器的深度学习模型(有四个)也一样放到vibe_data文件夹里。 (Still in the Python folder directory, I suggest not using the prepare_data.sh given by the author to download, just use the link inside to download to the browser, and the downloaded file is a compressed package. Then create a new data folder in the Vid2doppler directory, and then create a new vibe_data folder in it, and unzip the compressed package in vibe_data.Similarly, the deep learning models of the classifier (there are four) are also placed in the vibe_data folder.)**

七、最后，plot_synth_dop.py这个文件我被困扰了很久。因为我显卡的显存只有8g，所以需要限制transformer一次性使用全部显存，不然可能会报错。你们要是显存不够，可以照做。 (Finally, I was troubled by the plot_synth_dop.py file for a long time. Because my graphics card has only 8GB of video memory, I need to limit the transformer to use all the video memory at once, otherwise it will maybe report an error. If you don’t have enough video memory, you can do the same.)

.......................................................
import gc
.......................................................
.......................................................
def main(args):
    # Configure GPU memory growth - this prevents TF from taking all GPU memory at once
    gpus = tf.config.experimental.list_physical_devices('GPU')
    if gpus:
        try:
            for gpu in gpus:
                # Set memory growth option
                tf.config.experimental.set_memory_growth(gpu, True)
                # Optionally, you can limit GPU memory usage
                # Adjust the memory limit (in MB) based on your GPU
                memory_limit = 5632  # 5.5GB - adjust this based on your GPU memory
                tf.config.experimental.set_virtual_device_configuration(
                    gpu,
                    [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=memory_limit)]
                )
        except RuntimeError as e:
            print(e)
        ..................................................................
        ..................................................................

#将decoded = autoencoder.predict(synth_spec_test) decoded = decoded[:,:,:,0] decoded = rolling_window_combine(decoded)这三行更改为： (Change decoded = autoencoder.predict(synth_spec_test) decoded = decoded[:,:,:,0] decoded = rolling_window_combine(decoded) to :)

        ..................................................................
        ..................................................................
        # Process predictions in batches to reduce GPU memory usage
        batch_size = 8  # Adjust this based on your GPU memory
        num_batches = int(np.ceil(len(synth_spec_test) / batch_size))
        decoded = []

        for batch_idx in range(num_batches):
            start_idx = batch_idx * batch_size
            end_idx = min((batch_idx + 1) * batch_size, len(synth_spec_test))

            # Process batch
            batch_data = synth_spec_test[start_idx:end_idx]

            # Make prediction
            with tf.device('/GPU:0'):  # Ensure GPU usage
               batch_decoded = autoencoder.predict(batch_data, batch_size=batch_size)

            # Extract the results
            batch_decoded = batch_decoded[:,:,:,0]
            decoded.extend(batch_decoded)

            # Clear GPU memory after each batch
            tf.keras.backend.clear_session()

        decoded = np.array(decoded)
        decoded = rolling_window_combine(decoded)
        ..................................................................
        ..................................................................

把for idx in range(0, len(dop_spec_test)):这个循环改为： (Change the loop for idx in range(0, len(dop_spec_test)): to:）

        ..................................................................
        ..................................................................
        # Process video frames in batches
        frame_batch_size = 4  # Adjust based on your GPU memory
        total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
        processed_frames = 0
        while processed_frames < total_frames:
            batch_end = min(processed_frames + frame_batch_size, total_frames)

            frames = []
            for i in range(processed_frames, batch_end):
                ret, frame = cap.read()
                if not ret:
                    print(f"Warning: Failed to read frame {i}. Breaking loop.")
                    break

                # Process current frame
                original_synth = color_scale(
                    synth_spec_test[i],
                    matplotlib.colors.Normalize(vmin=0, vmax=np.max(synth_spec_test)),
                    "Initial Synthetic Doppler"
                )
                original_dop = color_scale(
                    dop_spec_test[i],
                    matplotlib.colors.Normalize(vmin=0, vmax=np.max(dop_spec_test)),
                    "Real World Doppler"
                )
                recon = color_scale(
                    decoded[i],
                    matplotlib.colors.Normalize(vmin=0, vmax=np.max(decoded)),
                    "Final Synthetic Doppler"
                )
                in_frame = color_scale(frame, None, "Input Video")

                # Combine frames
                output = np.hstack([in_frame, original_dop, original_synth, recon])
                out_vid.write(output)
                processed_frames += 1

                if processed_frames % 10 == 0:
                    print(f"Processed frames: {processed_frames}/{total_frames}")

                # Clear GPU memory if necessary
                if i % 32 == 0:  # Clear every 32 frames
                    tf.keras.backend.clear_session()

        cap.release()
        out_vid.release()
        print(f"Video processing complete. Total frames processed: {processed_frames}")
..................................................................
..................................................................

接着，我不确定这一步是否有效。你可能需要更改fps和TIME_CHUNK。原作者给的是24和3，我的视频是30帧，那么我的TIME_CHUNK就要改为2.4。如果你要传入60帧的视频，TIME_CHUNK就改为1.2。总之就是保证这两者相乘等于72，方便后面的视频合成。 (Next, I’m not sure if this step is effective. You may need to change fps and TIME_CHUNK.The original author gave 24 and 3, my video is 30 frames, so my TIME_CHUNK should be changed to 2.4. If you want to pass in a 60-frame video, TIME_CHUNK should be changed to 1.2. In short, make sure that the multiplication of these two equals 72, which is convenient for the subsequent video synthesis.)

    fps = 30
    TIME_CHUNK = 2.4

最后的最后，你要看到最终效果，一定要传入doppler_ge.npy文件，其次保证npy文件和视频放在同一目录下，不然的话plot_synth_dop.py给出的合成视频只有258字节，根本用不了。 (Finally, if you want to see the final effect, you must import the doppler_ge.npy file, and secondly, make sure that the npy file and the video are placed in the same directory, otherwise the synthetic video given by plot_synth_dop.py will only be 258 bytes, which is not usable at all.)

八，运行成功！(Run!) python doppler_from_vid.py --input_video /home/xan/Vid2Doppler/sample_video/Cycling/video.mp4 --model_path /home/xan/Vid2Doppler/data/vibe_data/ --doppler_gt --visualize_mesh 你应该会看到这些（You can see this）

[YueJiang-nj]

Thanks so much for this tutorial!