ValueError occurs while executing the HairGrow.py step.

@KeyuWu-CS I encountered a problem almost at the end of my process and need some assistance. 😭😭 While running the final command on my short male hair dataset, python HairGrow.py --yaml=configs/reconstruct/male_short, I encountered the following error. I tried to resolve it, but couldn't pinpoint the exact issue. Do you have any suggestions or insights on what might be causing this problem? Your advice would be greatly appreciated! Thank you! :)

Error log

setting configurations...
loading configs/reconstruct/base.yaml...
loading configs/reconstruct/test_maleB.yaml...
* HairGenerate:
   * connect_dot_threshold: 0.85
   * connect_scalp: True
   * connect_segments: True
   * connect_threshold: 0.005
   * connect_to_guide: None
   * dist_to_root: 6
   * generate_segments: True
   * grow_threshold: 0.9
   * out_ratio: 0.0
* PMVO:
   * conf_threshold: 0.1
   * filter_point: True
   * genrate_ori_only: None
   * infer_inner: True
   * num_sample_per_grid: 4
   * optimize: True
   * patch_size: 5
   * threshold: 0.05
   * visible_threshold: 1
* bbox_min: [-0.32, -0.32, -0.24]
* bust_to_origin: [0.006, -1.644, 0.01]
* camera_path: camera/calib_data/wky07-22/cam_params.json
* check_strands: True
* cpu: None
* data:
   * Conf_path: conf
   * Occ3D_path: ours/Occ3D.mat
   * Ori2D_path: best_ori
   * Ori3D_path: ours/Ori3D.mat
   * bust_path: Bust/bust_long.obj
* case: test_maleB
   * conf_threshold: 0.4
   * depth_path: render_depth
   * frame_interval: 2
   * image_size: [1920, 1080]
   * mask_path: hair_mask
   * raw_points_path: ours/colmap_points.obj
   * root: data
   * scalp_path: ours/scalp_tsfm.obj
   * strands_path: ours/world_str_raw.dat
* device: cuda:0
* gpu: 0
* image_camera_path: ours/cam_params.json
* infer_inner:
   * render_data: True
   * run_mvs: True
* name: 10-16
* ngp:
   * marching_cubes_density_thresh: 2.5
* output_root: output
* prepare_data:
   * fit_bust: True
   * process_bust: True
   * process_camera: True
   * process_imgs: True
   * render_depth: True
   * run_ngp: True
   * select_images: True
* save_path: refine
* scalp_diffusion: None
* seed: 0
* segment:
   * CDGNET_ckpt: assets/CDGNet/LIP_epoch_149.pth
   * MODNET_ckpt: assets/MODNet/modnet_photographic_portrait_matting.ckpt
   * scene_path: None
* vsize: 0.005
* yaml: configs/reconstruct/test_maleB
existing options file found (different from current one)...
17c17
<     optimize: null
---
>     optimize: true
override? (y/n) generate from scalp
voxel size: 192 256 256
100%|███████████████████████████████████████████████████████████| 60000/60000 [16:39<00:00, 60.00it/s]
num guide: 0
100%|██████████████████████████████████████████████████████████| 30503/30503 [00:58<00:00, 521.34it/s]
100%|█████████████████████████████████████████████████████████| 30503/30503 [00:20<00:00, 1493.68it/s]
done...
Smoothing strands: 100%|██████████████████████████████████████| 12108/12108 [00:08<00:00, 1396.04it/s]
connect segments...
100%|█████████████████████████████████████████████████████████| 12108/12108 [00:10<00:00, 1176.97it/s]
100%|██████████████████████████████████████████████████████████| 12108/12108 [01:00<00:00, 201.43it/s]
fail: 8151
done...
100%|███████████████████████████████████████████████████████| 12108/12108 [00:00<00:00, 297656.32it/s]
Smoothing strands: 100%|██████████████████████████████████████| 12108/12108 [00:10<00:00, 1171.83it/s]
num of strands: 12108
num of good strands: 0.0
connect poor strands to good strands...
iter: 0
num of good strands: 0
num of out strands: 0
current thr_dist: 0.5
current thr_dot: 0.9
Traceback (most recent call last):
  File "/MonoHair/HairGrow.py", line 963, in <module>
    connect_strands = HairGrowSolver.connect_to_scalp(strands,num_root)
  File "/MonoHair/HairGrow.py", line 655, in connect_to_scalp
    core_strands = np.concatenate(core_strands,0)
  File "<__array_function__ internals>", line 180, in concatenate
ValueError: need at least one array to concatenate

@KeyuWu-CS While running whole steps on two datasets, ksyusha1 (which you shared) and my_video (which I recorded myself in FHD(1920×1080), 60FPS using a smartphone camera), the issue did not occur with the ksyusha1 dataset, but it did occur with the my_video dataset. I used the same parameters for Bust_fit and reconstruction configs as in ksyusha1.

What could be the cause? Is there any part you might be suspicious about? 🤔🤔🤔

Traceback (most recent call last): File "/MonoHair/HairGrow.py", line 963, in connect_strands = HairGrowSolver.connect_to_scalp(strands,num_root) File "/MonoHair/HairGrow.py", line 655, in connect_to_scalp core_strands = np.concatenate(core_strands,0) File "<__array_function__ internals>", line 180, in concatenate ValueError: need at least one array to concatenate

@KeyuWu-CS I think that I have identified the reason for the error mentioned above. The key_frame.json file for 16 fixed view was created incorrectly.

Please see the figures below. The first set of images is the training_image samples you provided, and the second set is the training_image samples I created using instant_ngp with colmap features. As you can see, 16 fixed views I created are not perfectly centered, so that DeepMVSHair might not be able to train correctly. Creating perfectly centered 16 fixed view with instant_ngp is quite challenging without any supporting functions. Could you please explain how you achieved it?

training_image you provided:

training_image I made:

Sorry for later answer. For the error info, I think the scalp you create is away from 3D line map. You can visual the 3D line map and the corresponding scalp, make sure they are close. For the 16 fixed views. You can zoom in and out to adjust all the 16 views have complete hair. You can load base_cam.json and in Instant-NGP GUI to see the the position of the camera matrix. It is best if the subject is in the middle of the camera matrix.

@KeyuWu-CS First of all, I apologize for the inconvenience, but I am very eager to experience this SOTA image-based modeling. I would greatly appreciate it if you could offer me a bit more advice.

For the error info, you mentioned to check if the scalp I created and the 3D line map are sufficiently close. As far as I know, the following are the scalp and the 3D line map.
Is there a problem? And if there is a problem, what is the method to make them as close as possible? Maybe what I should do is improve the 16 fixed views, right?

/my_dataset/imgs/012
├── bust_depth.png                <--- scalp
├── bust_hair_depth.png  
├── hair_depth.png
├── mask.png
├── origin.png
└── undirectional_map.png      <--- 3D line map

For the 16 fixed views, you said you can zoom in and out of the 16 views. In this document, it only covers creating the front view key_frame.json among the 16 views. Should I need to take the key_frame.json 16 times? The sample you provided includes colmap/base_cam.json, but following the document's method, colmap/base_cam.json is not generated.
Could you explain how to generate and manually adjust the 16 views from the front key_frame.json?

When I can completely perform this process, I will support the document with additional images and descriptions, and I will respond diligently to other issues on your behalf. Thank you so much for your help. :)

From the given two images, It seems that the scalp is far away from the 3D line map. I suggest you visual 3D line map and scalp in open3D, In 3D space you can see their positional relationship more clearly. We provide some function in visualization.py. It's easy to visualize the scalp and 3D line map using Open3D, We provide you reference code:

import trimesh
import os
from visualization import *
def load_bust(path):
    bust = trimesh.load(path)
    vertices = np.array(bust.vertices)
    faces = np.array(bust.faces)
    normals = np.array(bust.vertex_normals)
    return vertices, faces,normals

root = "data/full/jenya2"
# root = r"E:/wukeyu/hair/project/MonoHair_release/data/full/jenya2"
output_path = os.path.join(root,"output","10-16")
print(output_path)

bust_path = os.path.join(root,"ours/bust_long_tsfm.obj")
vertices, faces,normals = load_bust(bust_path)
vertices += np.array([0.006, -1.644, 0.010])

select_points = np.load(os.path.join(output_path, "refine", "select_p.npy"))
select_ori = np.load(os.path.join(output_path, "refine", "select_o.npy"))
min_loss = np.load(os.path.join(output_path, "refine", "min_loss.npy"))

filter_unvisible_points = np.load(os.path.join(output_path,"refine","filter_unvisible.npy"))
filter_unvisible_ori = np.load(os.path.join(output_path,"refine","filter_unvisible_ori.npy"))
up_index =  filter_unvisible_ori[:,1]>0
filter_unvisible_ori[up_index]*=-1

index = np.where(min_loss <= 0.005)[0]  ### filter points with high loss, you can set differnt values for different cases.

select_ori = select_ori[index]
select_points = select_points[index]
print("num select", select_ori.shape[:])

reverse_index = select_ori[:, 1] > 0
select_ori[reverse_index] *= -1

select_ori_visual = np.abs(select_ori)
# select_ori_visual = (select_ori + 1) / 2

vis_points = vis_point_colud(select_points, select_ori_visual)
vis_bust = vis_mesh(vertices, faces)
vis_norm = vis_normals(select_points, select_ori * 0.004, select_ori_visual)
draw_scene([vis_points, vis_norm, vis_bust])
select_points = np.concatenate([select_points, filter_unvisible_points], 0)
select_ori = np.concatenate([select_ori, filter_unvisible_ori], 0)

reverse_index = select_ori[:, 1] > 0
select_ori[reverse_index] *= -1
select_ori_visual = np.abs(select_ori)
# select_ori_visual = (select_ori + 1) / 2

vis_fuse = vis_point_colud(select_points, select_ori_visual)
vis_fuse_norm = vis_normals(select_points, select_ori * 0.004, select_ori_visual)
draw_scene([vis_fuse, vis_fuse_norm,vis_bust])

In this step, you will see results similar to this: 1723540725891

For 16 fixed views, we have defined their relative pose in cam_params.json. You only need to manually create one key frame(front view). Then we will align key_frame with camera view 000 in cam_params.json. After this step, we can generate 16 fixed views in Instant-NGP coordinate system (That is base_cam.json). base_cam.json will be generated in prepare_data.py. But before this, you must create Key_frame.json

If scalp are far away from the 3D line map, you should check fit_bust step. This step will move SMPLX model close to 3D line map

@KeyuWu-CS Thank you for your advice. In the end, I ran the entire process.

While running this process, I encountered a few questions. I ran the whole process with the ksyusha1 dataset you provided using the same images and configuration. But, the results of instant-ngp are not the same. Please, see the images below. (The left one is yours, the right one is mine) Even though we use same images and config, your result is clearer and less noisy. I suspect that this discrepancy is due to the colmap process. How can I achieve the same quality of results in instant-ngp?

How many images are you used when running COLMAP, If remember correctly, I used about 300 images(as much as possible) to estimate the camera parameters. When running COLMAP, I set shared focal length and OpenCV model. 1723973121537

COLMAP using the following two step: 1723973222919 Then export .txt files. Note: the reconstructed results very rely on the instant-NGP results. Make sure have a normal initialization.

@KeyuWu-CS Thank you for your response! The Shared for all images option you told me surprisingly works well for me too! (see the reconstruction result below) I encountered an error when I used the dataset that I recorded of myself. I believe this is my last issue.

It's weird. When I ran the entire process, from making the reconstruction using COLMAP to the HairGrow.py step, the following errors occurred only with my dataset(289 images) I recorded of myself. I also ran the same process, including COLMAP and instant-ngp, with images from jenya2 and ksyusha1. However, these errors only occurred with my own dataset.

setting configurations...
loading configs/reconstruct/base.yaml...
loading configs/reconstruct/test_maleB_fix.yaml...
* HairGenerate:
   * connect_dot_threshold: 0.85
   * connect_scalp: True
   * connect_segments: True
   * connect_threshold: 0.005
   * connect_to_guide: None
   * dist_to_root: 6
   * generate_segments: True
   * grow_threshold: 0.9
   * out_ratio: 0.0
* PMVO:
   * conf_threshold: 0.1
   * filter_point: True
   * genrate_ori_only: None
   * infer_inner: True
   * num_sample_per_grid: 4
   * optimize: True
   * patch_size: 5
   * threshold: 0.05
   * visible_threshold: 1
* bbox_min: [-0.32, -0.32, -0.24]
* bust_to_origin: [0.006, -1.644, 0.01]
* camera_path: camera/calib_data/wky07-22/cam_params.json
* check_strands: True
* cpu: None
* data:
   * Conf_path: conf
   * Occ3D_path: ours/Occ3D.mat
   * Ori2D_path: best_ori
   * Ori3D_path: ours/Ori3D.mat
   * bust_path: Bust/bust_long.obj
   * case: test_maleB_fix
   * conf_threshold: 0.4
   * depth_path: render_depth
   * frame_interval: 2
* image_size: [1920, 1080]
   * mask_path: hair_mask
   * raw_points_path: ours/colmap_points.obj
   * root: data
   * scalp_path: ours/scalp_tsfm.obj
   * strands_path: ours/world_str_raw.dat
* device: cuda:0
* gpu: 0
* image_camera_path: ours/cam_params.json
* infer_inner:
   * render_data: True
   * run_mvs: True
* name: 10-16
* ngp:
   * marching_cubes_density_thresh: 2.5
* output_root: output
* prepare_data:
   * fit_bust: True
   * process_bust: True
   * process_camera: True
   * process_imgs: True
   * render_depth: True
   * run_ngp: True
   * select_images: True
* save_path: refine
* scalp_diffusion: None
* seed: 0
* segment:
   * CDGNET_ckpt: assets/CDGNet/LIP_epoch_149.pth
   * MODNET_ckpt: assets/MODNet/modnet_photographic_portrait_matting.ckpt
   * scene_path: None
* vsize: 0.005
* yaml: configs/reconstruct/test_maleB_fix
existing options file found (different from current one)...
17c17
<     optimize: null
---
>     optimize: true
override? (y/n) generate from scalp
voxel size: 192 256 256
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 60000/60000 [07:47<00:00, 128.47it/s]
num guide: 0
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 149281/149281 [03:35<00:00, 691.62it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 149281/149281 [01:28<00:00, 1690.19it/s]
done...
Smoothing strands: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 55672/55672 [00:32<00:00, 1699.85it/s]
connect segments...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 55672/55672 [00:44<00:00, 1262.49it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 55672/55672 [05:52<00:00, 157.81it/s]
fail: 16322
done...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 55672/55672 [00:00<00:00, 369613.68it/s]
Smoothing strands: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 55672/55672 [00:39<00:00, 1403.57it/s]
num of strands: 55672
num of good strands: 0.0
connect poor strands to good strands...
iter: 0
num of good strands: 0
num of out strands: 0
current thr_dist: 0.5
current thr_dot: 0.9
Traceback (most recent call last):
  File "/MonoHair/HairGrow.py", line 963, in <module>
    connect_strands = HairGrowSolver.connect_to_scalp(strands,num_root)
  File "/MonoHair/HairGrow.py", line 655, in connect_to_scalp
    core_strands = np.concatenate(core_strands,0)   
  File "<__array_function__ internals>", line 180, in concatenate
ValueError: need at least one array to concatenate

I checked the HairGrowSolver.connect_to_scalp function in HairGrow.py, where the issue seems to arise, but I cannot figure out why an empty connect_strands is being generated.

At First, I thought it might be because my hair is too short or there was an issue with the preprocessing steps I performed, but when I manually processed the jenya2 images, which have a hair length similar to mine, the output was successfully generated.

Is there a part of the configure settings that I should set differently when using my dataset compared to the provided jenya2 config settings? As you can see below, I confirmed that there are no issues with the 16 fixed viewpoint centered, scalps and 3D line map matched, landmark2d generated correctly that you advised me to check. If you have any ideas, please let me know.

3D Line maps and Scalps

I think I found what the problem is. My scalp and hair are still far apart. This is weird because I used the same images (jenya2) as you, the same colmap settings, and generated using the same instant-ngp with the 16 fix view perfectly centered. Why are my scalp and hair still separated? I think there might be changes in other submodules. Do you have any idea why this still might be happening? How can I accurately bring my scalp and hair closer together?

@KeyuWu-CS To assist in understanding, I have organized the results of the jenya2 image as I generated and the results you provided. Both results were generated from the same image.

Here are training_images I generated:

Here are training_images you provided:

Here are imgs/bust_hair_depth images I generated:

Here are imgs/bust_hair_depth images you provided:

Here are imgs/undirectional_map images I generated:

Here are imgs/undirectional_map images you provided:

The following are the settings I used for generation. These are identical to the provided sample. recontruct config:

_parent_: configs/reconstruct/base.yaml

data:
  case: test_jenya2
  image_size: [1920,1080]
  frame_interval: 2
  conf_threshold: 0.4
prepare_data:
  fit_bust: true

PMVO:
   patch_size: 5
   conf_threshold: 0.1

ngp:
  marching_cubes_density_thresh: 2.5

HairGenerate:
  connect_threshold: 0.005
  grow_threshold: 0.9
  connect_dot_threshold: 0.85
  out_ratio: 0.

Bust_fit config:

_parent_: configs/Bust_fit/base.yaml

subject: test_jenya2

savepath: data
subject_path: data/test_jenya2
camera_path: data/test_jenya2/ours/cam_params.json

data:
    image_size: [1920,1080]

smplx:
    n_shape: 300
    n_exp: 100

loss:
  scale_weight: 1

Which parts and how should I adjust to make the scalp and hair fit accurately?

This issue happened not related to the 16 views generation process, but only related to the bust_fit step. In this step, the template smplx model will be aligned near the hair. Please make sure to use/ours/bust_long_tsfm.obj. And confirm that the result of multiview optimization is reasonable. You can check the visualization results in the /data/jenya2/optimize/vis directory.

@KeyuWu-CS Yes, that seems to be the problem. When I look at optimize/vis, my results are optimized using only a single image. I think this is because only one landmark2d is being generated. Has the code related to this changed in any way? I cannot determine from the code why there is always only one landmark2d. iris:

iris
├── 0.png
├── 0.txt
├── 1110.png
├── 1110.txt
├── 115.png
├── 115.txt
├── 120.png
├── 120.txt
├── 135.png
├── 135.txt
├── 150.png
├── 150.txt
├── 175.png
├── 175.txt
├── 1790.png
├── 1790.txt
├── 180.png
├── 180.txt
├── 1880.png
├── 1880.txt
├── 1890.png
├── 1890.txt
├── 1920.png
├── 1920.txt
├── 1930.png
├── 1930.txt
├── 195.png
├── 195.txt
├── 1965.png
├── 1965.txt
├── 1966.png
├── 1966.txt
├── 1995.png
├── 1995.txt
├── 2010.png
├── 2010.txt
├── 2025.png
├── 2025.txt
├── 202.png
├── 202.txt
├── 2045.png
├── 2045.txt
├── 2070.png
├── 2070.txt
├── 216.png
├── 216.txt
├── 225.png
├── 225.txt
├── 232.png
├── 232.txt
├── 255.png
├── 255.txt
├── 258.png
├── 258.txt
├── 285.png
├── 285.txt
├── 297.png
├── 297.txt
├── 300.png
├── 300.txt
├── 30.png
├── 30.txt
├── 325.png
├── 325.txt
├── 332.png
├── 332.txt
├── 345.png
├── 345.txt
├── 370.png
├── 370.txt
├── 375.png
├── 375.txt
├── 390.png
├── 390.txt
├── 417.png
├── 417.txt
├── 420.png
├── 420.txt
├── 465.png
├── 465.txt
├── 475.png
├── 475.txt
├── 500.png
├── 500.txt
├── 510.png
├── 510.txt
├── 540.png
├── 540.txt
├── 550.png
├── 550.txt
├── 560.png
├── 560.txt
├── 56.png
├── 56.txt
├── 580.png
├── 580.txt
├── 600.png
├── 600.txt
├── 60.png
├── 60.txt
├── 610.png
├── 610.txt
├── 85.png
├── 85.txt
├── 95.png
└── 95.txt

landmark2d:

landmark2d
├── 0.png
└── 0.txt

optimize/viz/009999.png:

Can you please check the generate_landmark2d function. Set some output to check why only one image has landmark2D. It's weired, but it must the problem. I have tested serval days ago, don't face the problem.

This issue happened not related to the 16 views generation process, but only related to the bust_fit step. In this step, the template smplx model will be aligned near the hair. Please make sure to use/ours/bust_long_tsfm.obj. And confirm that the result of multiview optimization is reasonable. You can check the visualization results in the /data/jenya2/optimize/vis directory.

I already checked that part. Your insight that it wasn't a 16 fix view issue was incredibly helpful. The real problem was that the code running in the Docker container was creating files without write permissions. I didn't initially think it was a permissions issue because one file was being generated while other were not. I've finished debugging the generate_landmark2d function and now running t he inference again. I'm confident it will work this time! Thank you soooo much!!

KeyuWu-CS / MonoHair

ValueError occurs while executing the HairGrow.py step. #13