Generate UV mapping from a mesh constructed from Point Cloud?

[lthiet]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

My Question

Hi!

My pipeline looks like this:

• 1. Take multiple pictures from a scene in the real world
• 2. Use OpenSfM to generate a point cloud
• 3. Use Open3D to convert the point cloud to a mesh
• 4. Generating UV mapping from the mesh <-- this is where I am stuck
• 5. Export the resulting mesh file to Unity and render it

Regarding step 3, the input was a set of point, and with each point was assigned a color and a normal vector. I ran this code:

pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(points)
pcd.colors = o3d.utility.Vector3dVector(colors / 255.0)
pcd.normals = o3d.utility.Vector3dVector(normals)

mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson(pcd, depth=13, scale=1.1, linear_fit=False)[0]
bbox = pcd.get_axis_aligned_bounding_box()
mesh_cropped = mesh.crop(bbox)
mesh_cropped.compute_vertex_normals()

And the resulting visualization looks like this:

My question is: is there any way to generate a UV map from this mesh and the associated 2d texture with it?

[theNded]

Unfortunately, we don't have this functionality yet. It was on our task list but we did not have enough bandwidth for the feature. As of now you may have to use blender for the task.

[lthiet]

I see. Would you already know to perform such a task in Blender?

[theNded]

You may refer to their manual to generate a uv map. I used it a while ago but do not remember the details. But color mapping is still nontrivial -- we had an experimental branch several years ago but did not move forward (link, just for the record, not directly usable).

@yxlao we may want to incorporate this library since uv map has been frequently requested. https://github.com/MozillaReality/xatlas-web (AS IS license) and add our experimental color map optimization on texture as well.

Update: additional references: https://github.com/mworchel/xatlas-python

[lthiet]

Thanks a lot for the references! I will have a look at them and see if I managed to write a pipeline for generating UV maps given vertex colors.

[LemonPi]

I had a similar need of programmatically generating mesh textures given vertex colors and resorted to MeshLab (pymeshlab). You can do something like:

import pymeshlab
m = pymeshlab.Mesh(points, v_normals_matrix=normals, v_color_matrix=colors / 255.0) # color is N x 4 with alpha info
ms = pymeshlab.MeshSet()
ms.add_mesh(m, "pc_scan")
ms.generate_surface_reconstruction_screened_poisson(depth=13, scale=1.1)
# not familiar with the crop API, but I'm sure it's doable
# now we generate UV map; there are a couple options here but this is a naive way
ms.compute_texcoord_parametrization_triangle_trivial_per_wedge()
# create texture using UV map and vertex colors
ms.compute_texmap_from_color(textname=f"my_texture_name") # textname will be filename of a png, should not be a full path
# texture file won't be saved until you save the mesh
ms.save_current_mesh(mesh_path)

[BenjaminHelyer]

I also recently ran into this problem. The solution provided by @LemonPi worked for my needs. You may need to tweak the file format (I used .obj) and the UV map generation function.

Unfortunately the pymeshlab functionality is quite slow - this solution extended my runtime 3-4x. So, I think the feature request for Open3D should still hold, even if it's low priority given that there is a decent workaround.

[nguyenquocduongqnu]

I had a similar need of programmatically generating mesh textures given vertex colors and resorted to MeshLab (pymeshlab). You can do something like:

import pymeshlab
m = pymeshlab.Mesh(points, v_normals_matrix=normals, v_color_matrix=colors / 255.0) # color is N x 4 with alpha info
ms = pymeshlab.MeshSet()
ms.add_mesh(m, "pc_scan")
ms.generate_surface_reconstruction_screened_poisson(depth=13, scale=1.1)
# not familiar with the crop API, but I'm sure it's doable
# now we generate UV map; there are a couple options here but this is a naive way
ms.compute_texcoord_parametrization_triangle_trivial_per_wedge()
# create texture using UV map and vertex colors
ms.compute_texmap_from_color(textname=f"my_texture_name") # textname will be filename of a png, should not be a full path
# texture file won't be saved until you save the mesh
ms.save_current_mesh(mesh_path)

Can you give me a code example of converting an obj file into a UV mapping image? @BenjaminHelyer

[benjaminum]

Open3D has integrated UVAtlas in 0.16. See http://www.open3d.org/docs/latest/python_api/open3d.t.geometry.TriangleMesh.html#open3d.t.geometry.TriangleMesh.compute_uvatlas

Closing this feature request. The discussion can continue here or at https://github.com/isl-org/Open3D/discussions .

[nam-usth]

I can suggest another way to create UV atlas by using MeshLab (pymeshlab) and a library named xatlas.

Before we go, this command helps install the xatlas

pip install xatlas

Once ready, we can use the following code to generate UV atlas:

import numpy as np
import open3d as o3d
import os
import pymeshlab
import trimesh
import xatlas

# %% Point cloud

def read_pcd(file):
    # Try open and read the file
    try:
        pcd = o3d.io.read_point_cloud(file, format='xyzn')
        pcd = pcd.voxel_down_sample(voxel_size=0.075)
        return pcd
    except:
        print("Couldn't read file")

def convert_pcd_to_mesh(pcd):
    try:
        mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson(pcd, depth=9)
        return mesh
    except:
        print("Invalid Data...")

# %% Mesh

def read_mesh(file):
    try:
        mesh = o3d.io.read_triangle_mesh(file)
        return mesh
    except:
        print("Couldn't read file")

# %% Input data handler

def input_handler(file, mode="mesh"):
    if (mode == 'point_cloud'):
        data = convert_pcd_to_mesh(read_pcd(file))

    if (mode == 'mesh'):
        data = read_mesh(file)
    return data

# %% Convert mesh from open3d to trimesh and vice versa

def o3d_trimesh(o3d_mesh):
    tri_mesh = trimesh.Trimesh(np.asarray(o3d_mesh.vertices), np.asarray(o3d_mesh.triangles), 
                               vertex_normals=np.asarray(o3d_mesh.vertex_normals))   
    return tri_mesh

def trimesh_o3d(tri_mesh):
    # Convert vertices and faces from trimesh to open3d format
    vertices = o3d.utility.Vector3dVector(tri_mesh.vertices)
    faces = o3d.utility.Vector3iVector(tri_mesh.faces)

    # Create an open3d TriangleMesh object
    o3d_mesh = o3d.geometry.TriangleMesh(vertices, faces)

    return o3d_mesh

# %% Main function

if __name__ == "__main__":
    CWD_PATH = os.getcwd()
    DS_NAME = 'brick'
    FILE_NAME = 'brick_part01.pcd'

    file = os.path.join(CWD_PATH, 'dataset', DS_NAME, FILE_NAME)
    mesh_data = input_handler(file, 'point_cloud')

    if (type(mesh_data) is tuple):
        mesh = mesh_data[0]
    else:
        mesh = mesh_data

    # Convert o3d to trimesh
    tri_mesh = o3d_trimesh(mesh)

    # Export the parametrized mesh with xatlas
    vmapping, indices, uvs = xatlas.parametrize(tri_mesh.vertices, tri_mesh.faces)
    xatlas.export(FILE_NAME[:-4] + ".obj", tri_mesh.vertices[vmapping], indices, uvs)

    # Import the xatlas saved mesh back to pymeshlab to compute curvature and build an image from the UV atlas
    ms = pymeshlab.MeshSet()
    ms.load_new_mesh(FILE_NAME[:-4] + '.obj')

    curv = ms.compute_scalar_by_discrete_curvature_per_vertex(curvaturetype=1)

    ms.compute_texmap_from_color(textname='K_' + FILE_NAME[:-4] + '_additional_info.png')
    ms.save_current_mesh('temp.ply')

P/S: The point cloud data was obtained at this link. You can download the raw point cloud then rename the extension of the file to .pcd

[gojushin]

@nam-usth You don't need the extra trimesh dependency. The unwrapping part can all be done directly in open3d:

import open3d as o3d
import numpy as np
import xatlas

def generate_uv_atlas(mesh: open3d.cpu.pybind.geometry.TriangleMesh): -> open3d.cpu.pybind.geometry.TriangleMesh:
    # Get Mesh Properties as numpy array
    vertices = np.asarray(mesh.vertices)
    triangles = np.asarray(mesh.triangles)

    # Unwrap
    vmapping, indices, uvs = xatlas.parametrize(vertices, triangles)

    # Reorder UVs to represent o3d structure
    reordered_uvs = uvs[indices.flatten()].astype(np.float64)

    # Assemble new o3d mesh
    new_mesh = o3d.geometry.TriangleMesh()

    new_mesh.vertices = o3d.utility.Vector3dVector(vertices[vmapping])
    new_mesh.triangles = o3d.utility.Vector3iVector(indices)
    new_mesh.triangle_uvs = o3d.utility.Vector2dVector(reordered_uvs)

    # Optional
    new_mesh.merge_close_vertices(eps=0.001)
    new_mesh.compute_vertex_normals()

    return new_mesh

I am sure the function can still be optimized, but this should convey the rough idea.