## ❓ Is it possible to use pytorch3d with point cloud, different data maps and custom loss

[ttsesm]

Hi,

Thank you for the library and the support you provide here.

I am interested to see whether I can use pytorch3d for my study. I have a bunch of point clouds as csv files where each file contain 5 columns:

1. the x, y, z coordinates of the points,
2. the information of which points are emitting energy (emitting points) and the energy intensity
3. the actual energy value in each point which is the result of the energy emitted from the points in 2 plus the interreflected energy from the other points (diffusive points)

For the sake of simplicity and without considering any complex parameters in the loss (e.g. reflectance factor of each point, visibility and occlusion factor between the points, etc) I wanted to ask if it is possible to establish a simple loss function || r - ^r || in pytorch3d, where r is my "ground truth" energy values and ^r are the predicted ones and whether loading different data maps (i.e. in my case the "ground truth" energy values instead of rgb values) is feasible.

I was looking in the repository and I found that you support rendering point clouds (https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/render_colored_points.ipynb), so in principle I could load my points, then use the features=[] attribute in point_cloud = Pointclouds(points=[verts], features=[rgb]) to insert my own mappings (I guess, please correct me if I am wrong here) but I am not sure how to create my own custom loss if possible. Thus, I would appreciate if you could provide me with some feedback whether what I am asking is achievable with pytorch3d, and if yes how (pointing me possibly to a tutorial which I might have overlooked or whatever).

Thanks.

[nikhilaravi]

@ttsesm You can use the features to store additional information about each point - it doesn't matter what is stored so long as it is has one value per point in points e.g. if points is of shape (N, P, 3) then features has to be of shape (N, P, D) where D can be anything. In your case it sounds like features should be the energy value in point 3.

Regarding the loss you can define this separately and it isn't related to how the PyTorch3D renderer is implemented.

[albertotono]

Hi @nikhilaravi, Is it possible to import file from PartNet with the .ply + .txt + .pts format in pytorch3d and render them accordingly with different colors? Is there a method already developed and available?

PartNet unfortunately uses Blender, it would be great to move PartNet with Pytorch3d.
Best Regards

[ttsesm]

@nikhilaravi thanks for the prompt response and the insightful feedback. Ok, I will try to apply a test case scenario and see how it works and possibly report back.

[nikhilaravi]

@albertotono currently for pointclouds we only support reading in .ply files.

[ttsesm]

@nikhilaravi if I want to load multiple pointclouds of different sizes how it can be done. I've tried to do it with the following code snippet:

    verts = torch.Tensor(pcds).to(device)
    illum = torch.Tensor(lx).to(device)
    point_cloud = Pointclouds(points=[verts], features=[illum])

where pcds is a list of pointcloud vertices as numpy arrays of different size and illum the corresponding features list as numpy arrays for each point in each cloud. However, I am getting the following error:

    raise ValueError("Clouds in list must be of shape Px3 or empty")
ValueError: Clouds in list must be of shape Px3 or empty

What I am doing is padding with zerros the missing indixes of the smallest pcds in regards to the biggest in size pcd (I am not sure though whether this is necessary though) then all my pcds are of size [6, 20000, 3] but still it complaints, lx is of size [6, 20000, 1]. Any idea what I might be doing wrong?

[bottler]

You describe multiple attempts and it isn't entirely clear what you mean. There are basically two ways to initialise a Pointclouds object. 1) Either with a single tensor (not in a list) as the points and a single tensor as the features, which might each have shape [6, 20000, 3] - useful when all the pointclouds are the same size, or 2) as a list of pointclouds which each have a shape like (something, 3).

If pcds is a list of pointclouds which are numpy arrays of shape (something, 3), and similar for lx as the colors then you should do

points = [torch.Tensor(p).to(device) for p in pcds]
features = [torch.Tensor(x).to(device) for x in lx]
point_cloud = Pointclouds(points=points, features=features)

By the way, when you write something like Pointclouds(points=[verts], features=[illum]) you are sending one-element lists to the constructor, which means you are telling pytorch3d that you want to create a batch of one point cloud. You don't mean to do this.

[ttsesm]

@bottler thanks for the feedback. Option 2 worked finally without issues.

[ttsesm]

Are there any tutorials or sample code which shows how I can use the Pointclouds() structure to regress/predict my features which is my energy on each point?

[ttsesm]

Hi, I am trying now to set an optimizer for estimating my features values from my given pointcloud. I have the following code snippet which fails though:

    verts = [torch.Tensor(p).to(device) for p in pcds]

    illum = [torch.Tensor(x).to(device) for x in lx]

    trg_point_clouds = Pointclouds(points=verts, features=illum)

    src_point_clouds = Pointclouds(points=verts)

    # We want to learn to estimate the source point cloud vertices energy
    # The shape of the estimated energy values is equal to the total number of vertices in trg_mesh
    _illum = torch.full(trg_point_clouds.features_packed().shape, 0.0, device=device, requires_grad=True)

    # The optimizer
    optimizer = torch.optim.SGD([_illum], lr=1.0, momentum=0.9)

    n_iter = 2000
    epoch = tqdm(range(n_iter))

    loss_f = torch.nn.L1Loss()
    losses = []

    for i in epoch:
        # Initialize optimizer
        optimizer.zero_grad()

        # Update of features
        src_point_clouds.features_packed = _illum

        # We compare the two sets of pointclouds by computing the L1 loss
        loss = loss_f(trg_point_clouds, src_point_clouds)

        # Print the losses
        epoch.set_description('total_loss = %.6f' % loss)

        # Save the losses for plotting
        losses.append(loss)

        # Optimization step
        loss.backward()
        optimizer.step()

My guess is that I doing wrong the src_point_cloud initialization and the update of the features list. I was checking on the tutorial with the fit mesh but I couldn't understand how to adjust it for my point cloud use case.

[bottler]

src_point_clouds.features_packed is a method. Overwriting it with a tensor doesn't make sense.

For this type of thing, don't try to modify the values in a Pointclouds object. Just create a new Pointclouds object with the new feature values on every iteration.

[ttsesm]

@bottler thanks for the feedback. Could you please elaborate a bit more how to do that, if possibly give an example. I was trying to find some samples but apparently there are none related to Pointclouds and feature values. I mean how do I pass the new feature values each time.

Also I am not sure whether I can compare the two point clouds in the way I am doing it by using the L1Loss since I am getting the following error:

    if not (target.size() == input.size()):
AttributeError: 'Pointclouds' object has no attribute 'size'

Which means that apparently I cannot compare the Pointclouds object directly with the torch loss.

Apologies for the newbie questions, I am quite new in torch as well as in pytorch3d and I am trying to understand the procedure to be followed.

[ttsesm]

Ok, playing around I've managed to get something running with the following snippet:

    verts = [torch.Tensor(p).to(device) for p in pcds]
    illum = [torch.Tensor(x).to(device) for x in lx]
    trg_point_clouds = Pointclouds(points=verts, features=illum)

    _illum = [torch.Tensor(x*0).to(device) for x in lx] # set tensor values to 0
    src_point_clouds = Pointclouds(points=verts, features=_illum)

    # We will learn to estimate the source point cloud vertices energy
    # The shape of the estimated energy values is equal to the total number of vertices in trg_mesh
    src_point_clouds.features_packed().requires_grad = True

    # The optimizer
    optimizer = torch.optim.SGD([src_point_clouds.features_packed()], lr=1.0, momentum=0.9)

    n_iter = 2000
    epoch = tqdm(range(n_iter))

    loss_f = torch.nn.L1Loss()
    losses = []

    for i in epoch:
        # Initialize optimizer
        optimizer.zero_grad()

        # We compare the two sets of pointclouds by computing the L1 loss
        loss = loss_f(trg_point_clouds.features_packed(), src_point_clouds.features_packed())

        # Print the losses
        epoch.set_description('total_loss = %.6f' % loss)

        # Save the losses for plotting
        losses.append(loss)

        # Optimization step
        loss.backward()
        optimizer.step()

the loss seems to be decreasing, slowly though, but as I see it, the optimization seems kind of irrelevant since as it is right now the pointcloud points (i.e. geometry) aren't really being used in the estimation of the features.

[bottler]

What I meant by using "a new Pointclouds object each iteration" is something like the following. I've taken your code, added some lines (all my added lines end in ##) and commented out some lines (always with ##). It's easier this way, respects the structure of Pointclouds object, and leaves the tensors to be optimised in your control.

verts = [torch.Tensor(p).to(device) for p in pcds]
illum = [torch.Tensor(x).to(device) for x in lx]
trg_point_clouds = Pointclouds(points=verts, features=illum)

## _illum = [torch.Tensor(x*0).to(device) for x in lx] # set tensor values to 0
## src_point_clouds = Pointclouds(points=verts, features=_illum)
_illum = [torch.Tensor(x*0, requires_grad=True).to(device) for x in lx] # set tensor values to 0 ##

## # We will learn to estimate the source point cloud vertices energy
## # The shape of the estimated energy values is equal to the total number of vertices in trg_mesh
## src_point_clouds.features_packed().requires_grad = True

# The optimizer
## optimizer = torch.optim.SGD([src_point_clouds.features_packed()], lr=1.0, momentum=0.9)
optimizer = torch.optim.SGD(_illum, lr=1.0, momentum=0.9) ##

n_iter = 2000
epoch = tqdm(range(n_iter))

loss_f = torch.nn.L1Loss()
losses = []

for i in epoch:
    # Initialize optimizer
    optimizer.zero_grad()

    src_point_clouds = Pointclouds(points=verts, features=_illum) ##

    # We compare the two sets of pointclouds by computing the L1 loss
    loss = loss_f(trg_point_clouds.features_packed(), src_point_clouds.features_packed())

    # Print the losses
    epoch.set_description('total_loss = %.6f' % loss)

    # Save the losses for plotting
    losses.append(loss)

    # Optimization step
    loss.backward()
    optimizer.step()

This code is your toy example. Your final question is a model design question - what are you really trying to achieve?

If you have source and target pointclouds where you know features in the target and you want to copy features to the source using the nearest target point(s) to each source point (which is basically what you are doing here, except you happen to know the points are lined up), then you can use our knn to do that.

[ttsesm]

@bottler thanks for the feedback. Briefly, what I would like achieve is to regress the features vector over some 3d points.

Imagine that I have multiple point clouds (eventually it might be possible to have also the reconstructed 3D mesh with vertices/faces/normals) of different size that are representing different room structures with their objects (i.e. beds, tables, chairs, etc). For each point I have the ground truth illuminance (as lux values), that is my features vector, and I also know which of these points correspond to light sources of the scene. So my data input are .csv files as a Nx5 array for each point cloud where each of the columns correspond to the label 0 or 1 if the point is a light source (for this I could also have the light intensity), the x, y , z coordinates of the point and the lux values respectively.

So now I wanted to experiment and see if it is possible to learn to estimate the lux, "energy", values for a given pointcloud (as a regression output). Moreover, possibly I would need to somehow include the information about the light sources. Not sure what the most efficient way to do this would be. One way would be to just include an extra feature for each point in the point cloud that represents the intensity of light from that point ( 0 for all but the light source points). So I would have a Nx5 input and get and regress my pointwise energy as output predictions.

Initially I was thinking that I could do that we Pytorch3d but as I see it now it seems that this is not possible since it does not provide any model where I could be doing such a regression. Thus, I would need to create a new model or modify an existing one for that purpose I guess.

[gkioxari]

Hi @ttsesm Your issue here doesn't seem to be related to a PyTorch3D bug or concrete issue with the library. It's great that @bottler has helped you out with your questions but please note that we are not here to offer consultation for people's projects. It's quite unfortunate but we simply don't have the bandwidth to guide users how to do their research or project. So unless you have a concrete bug to report, I will be closing this issue.