About select_sampling_method

[bbertolucci]

Hi, I am trying to understand IntegratedPifu code and trying to adapt it to my dataset who is not Thuman one. I reach the select_sampling_method() in TrainDataset.py; and I think it's an important function to match the obj file with the image file. So I have different questions :

1)

        # note, this is the solution for when dataset is "THuman"
        # adjust sigma according to the mesh's size (measured using the y-coordinates)
        y_length = np.abs(np.max(mesh.vertices, axis=0)[1])  + np.abs(np.min(mesh.vertices, axis=0)[1] )
        sigma_multiplier = y_length/188

You wrote 188 but what does it represents exactly ? Is it an average y-length of all the Thuman models ? Is it the maximum y-length of all the Thuman models ? Or is it a random number ? On the 3 free models of Thuman, I calculate the y length to 169, 171 and 187

2) What is the compensation_factor = 4.0 ?

3) Is there a way to verify if my mesh are correctly oriented (in the same direction of the image) after applying the rotation (R) ?

4) Just a fix on mesh_util.py : verts, faces, normals, values = measure.marching_cubes_lewiner(sdf, 0.5) must be replaced by : verts, faces, normals, values = measure.marching_cubes(sdf, 0.5, method="lewiner")

Thanks, and thank you also for your work it's so instructive !

[kcyt]

Hi,

1. You can use other number instead of 188, we picked 188 arbitrarily as it is one of the first numbers that worked for us.
2. compensation_factor is to increase the number of sample points generated because, for instance, spatial sampling needs at least self.num_sample_inout sample points to be in each minibatch and the sample points in the minibatch has to be exactly 50% inside of the mesh and 50% outside of the mesh. There is thus a need to generate more sample points at the start and then filter the ones that we do not need later.
3. You can use the trimesh library (https://trimsh.org/trimesh.html) and download Meshlab to rotate or/and visualize the meshes.
4. Yes, the new version of the package requires this, thank you for bringing this up.

[bbertolucci]

Hi, thank you for you answer, It's ok I totally understood your code ! But well there are still a lot of hardcoded values... For example my dataset have obj files with a y max of "2" and not "188" (like Thuman), resulting the displacement of inner points is too strong and push the points outside ! :p. In the same way depending on my dataset the depthmap value were too small too.. Well anyway I am currently regenerating my dataset to scale all of them to a max size of 188. It should work !

Thanks for the library, chatgpt gave me also some advise. I am sharing it : I added it to the end of : select_sampling_method()

            mesh.visual = trimesh.visual.ColorVisuals(mesh, face_colors=[255, 0, 0, 25])
            scene = trimesh.Scene(mesh)

            def create_grid(size, spacing):
                lines = []
                for i in np.arange(-size/2, size/2 + spacing, spacing):
                    lines.append(trimesh.creation.cylinder(radius=0.01, segment=[(-size/2, 0, i), (size/2, 0, i)], cap_ends=False))
                    lines.append(trimesh.creation.cylinder(radius=0.01, segment=[(i, 0, -size/2), (i, 0, size/2)], cap_ends=False))
                return lines

            grid = create_grid(size=200, spacing=10)
            for line in grid:
                scene.add_geometry(line, geom_name="grid")

            start_points_geom = trimesh.PointCloud(surface_points[:num_of_way_inside_pts], colors=[0, 255, 0, 255])
            scene.add_geometry(start_points_geom, geom_name="test")
            idx = 0
            for point, direction in zip(surface_points, normal_vectors):
                ray_points = point + direction * 20
                ray_geom = trimesh.creation.cylinder(radius=0.1, segment=[point, ray_points], cap_ends=False, vertex_colors=[0, 0, 255, 255])
                scene.add_geometry(ray_geom, geom_name="vector{}".format(idx))
                idx += 1

                if idx > num_of_way_inside_pts:
                    break

            scene.show()

            for i in range(num_of_way_inside_pts+1):
                scene.delete_geometry("vector{}".format(i))

            idx = 0
            for point, direction in zip(surface_points, z_displacement):
                ray_points = point + direction * 20
                ray_geom = trimesh.creation.cylinder(radius=0.1, segment=[point, ray_points], cap_ends=False, vertex_colors=[0, 0, 255, 255])
                scene.add_geometry(ray_geom, geom_name="displacement{}".format(idx))
                idx += 1

                if idx > num_of_way_inside_pts:
                    break

            scene.show()
            scene.delete_geometry("test")

            for i in range(num_of_way_inside_pts+1):
                scene.delete_geometry("displacement{}".format(i))

            start_points_geom = trimesh.PointCloud( surface_points_with_normal_sigma, colors=[255, 255, 0, 255])
            scene.add_geometry(start_points_geom, geom_name="test")

            scene.show(point_size=1)

            scene.delete_geometry("test")

            inside_points_geom = trimesh.PointCloud(way_inside_pts, colors=[255, 0, 0, 255])
            scene.add_geometry(inside_points_geom)

            inside_points_geom = trimesh.PointCloud(surface_points[0:num_of_way_inside_pts], colors=[0, 255, 255, 255])
            scene.add_geometry(inside_points_geom)

            # Ajouter les points intérieurs (en green) à la scène
            inside_points_geom = trimesh.PointCloud(outside_surface_points[0:num_of_way_inside_pts], colors=[0, 255, 0, 255])
            scene.add_geometry(inside_points_geom)

            idx = 0
            for inside, outside in zip(way_inside_pts, outside_surface_points):
                if not np.all(inside == 0) and not np.all(outside == 0):
                    ray_geom = trimesh.creation.cylinder(radius=0.1, segment=[inside, outside], cap_ends=False, vertex_colors=[0, 0, 255, 255])
                    scene.add_geometry(ray_geom, geom_name="displacement{}".format(idx))
                idx += 1

                if idx > num_of_way_inside_pts:
                    break

            scene.show(point_size=1)

It gave me a render like this :