Implement Object movement in the scene

[andrefdre]

I will now focus on implementing objects moving in the scene when the camera moves:

• [x] Find if object are inside the mesh (Found this http://www.open3d.org/docs/latest/tutorial/geometry/distance_queries.html also found this, which is a library we already use https://trimsh.org/trimesh.proximity.html)
• [x] Generate random poses for objects
• [x] Check if objects are touching the ground
• [ ] Move objects in a path

[andrefdre]

I manage to generate random points and perform a check to see if the point is inside the mesh. I used result = self.mesh_collision.contains(point) from trimesh and I based on this post mikedh/trimesh#241.

Short video of a random cube being moved in the scene:

https://user-images.githubusercontent.com/58526188/222509410-dfb95bbd-4988-4e5c-9657-1e3729dc945b.mp4

Right now it's only using the point of the pose, so part of the cube is inside the walls. I could also send an array of points to .contains() so in this case I could send the 4 corners.

[miguelriemoliveira]

Hi @andrefdre ,

I manage to generate random points and perform a check to see if the point is inside the mesh. I used result = self.mesh_collision.contains(point) from trimesh and I based on this post https://github.com/mikedh/trimesh/issues/241.

So you are using trimesh, not open3d?

congrats. The video looks good.

Right now it's only using the point of the pose, so part of the cube is inside the walls. I could also send an array of points to .contains() so in this case I could send the 4 corners.

I am not sure how long it takes, but the best would be this: The object you want to add to the scene is also a mesh, so test all the triangles to see if all are inside the room.

If this takes too long because the object mesh has too many vertices, you can create a convex hull mesh of the object and then test only the fewer vertices of the convex hull.

[andrefdre]

So you are using trimesh, not open3d?

Yes I decided to use trimesh since we already use it to check collision.

I am not sure how long it takes, but the best would be this: The object you want to add to the scene is also a mesh, so test all the triangles to see if all are inside the room.

My idea was to do something similar to that. I thought about converting the mesh to a point cloud then perform a downsample and check the remaining points.

Thinking a bit ahead, about checking the ground, my idea was to get the distance to the mesh pointing downwards and for each object have a pre-defined distance of the centroid to the lowest part of the object. With this, subtract the excess distance and set the pose of the object touching the ground. Is this overthinking the problem?

Sorry for my absence, this week I will work hard on this.

[pmdjdias]

I do not see the point of converting a mesh to a point cloud! In most of cases I guess using the points of the mesh an test if they are inside the room as suggested by Miguel should solve the case... A simpler solution would be to use a convex hull or even the centroid and maximum distance to a vertice and then just check the distance between the centroid and the outside mesh. Anyway to start, the suggestion to check if all the vertices of the mesh are inside the room should work fine.

[pmdjdias]

For the ground (I guess moving an object of the ground), I would simply fix the Y coordinate (height) so that the object is on the ground and move the object only in the X,Y coordinates (supposing Z is up).

[andrefdre]

For the ground (I guess moving an object of the ground), I would simply fix the Y coordinate (height) so that the object is on the ground and move the object only in the X,Y coordinates (supposing Z is up).

I thought about that, but in santuario the floor is not always at the same z coordinate.

Anyway to start, the suggestion to check if all the vertices of the mesh are inside the room should work fine.

I will try, when I have results I will share.

[andrefdre]

I followed your suggestions and used a convex hull otherwise it was too slow. This are the results:

https://user-images.githubusercontent.com/58526188/223153131-062d4325-facc-4ae3-96b4-fe96e261455f.mp4

I don't know if with more objects it wouldn't become too slow.

[andrefdre]

I implemented random object placement, always touching the ground.

https://user-images.githubusercontent.com/58526188/223188528-d284868f-1f86-484e-8258-eb3f0903fea5.mp4

I don't understand why the chair doesn't always move, I confirmed it's generating the pose and going into the function that sets it. But it doesn't move, only after many tries, or I pause the simulation and give it time to move. Going to investigate more about this.

[miguelriemoliveira]

I implemented random object placement, always touching the ground.

This looks really nice. Congratulations.

Some ideas to make it better:

1. More models of humans, chairs and other plausible objects
2. Rotate the objects around the z axis (the human is always facing one direction)

I don't understand why the chair doesn't always move, I confirmed it's generating the pose and going into the function that sets it. But it doesn't move, only after many tries, or I pause the simulation and give it time to move. Going to investigate more about this.

ok, I remember the gazebo services would sometimes fail with no explanation why...

[andrefdre]

2. Rotate the objects around the z axis (the human is always facing one direction)

It is rotating, the first 3 poses it's possible to see, the last two were just coincidentally the same.

1. More models of humans, chairs and other plausible objects

I will take one more day looking into the gazebo service to see if I can get to something, and afterward I will do this. Right now it's very easy to add more objects, the code is already implemented in a way to accept as many objects as previously defined.

[miguelriemoliveira]

Great!

[andrefdre]

I figured out why the objects weren't moving. In the SDF file, they had the property static set to true.

https://user-images.githubusercontent.com/58526188/223504306-fe635794-0145-4dbf-8e13-34bbaafd6ed4.mp4

I noticed sometimes objects spawn on top of the table or close to the camera, what should I do about this?

[pmdjdias]

Regarding the camera, shoud be easy, you can define a threshold distance between object centroid and camera position and discard object that are below the threshold. For the table it is harder, unless you manually define an exclusion area around the table. In my opinion we shoudl leave this possibility, the netwrok shoud learn also with this info, altough it is not common to walk over tables ;-)!

[andrefdre]

Regarding the camera, shoud be easy, you can define a threshold distance between object centroid and camera position and discard object that are below the threshold.

I followed your suggestion and also take into account the camera position. I placed a threshold distance of 0.5.

Is it possible to have a meeting this week to discuss the results and next steps?

[miguelriemoliveira]

Sure. You advanced a lot. Friday 9h or 10h?

[pmdjdias]

On friday I I have a meeting at 9:30 (and 11, and 12 ;-)). 10:00 would be better for me.

[DanielCoelho112]

10h is good for me too.

[andrefdre]

10h is good for me.

[miguelriemoliveira]

Ok 10h

On Wed, Mar 8, 2023, 18:12 André Cardoso @.***> wrote:

10h is good for me.

— Reply to this email directly, view it on GitHub https://github.com/DanielCoelho112/synfeal/issues/94#issuecomment-1460629557, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACWTHVRZLYMW2J4CB4OXGMLW3DDYTANCNFSM6AAAAAAVMF5NR4 . You are receiving this because you commented.Message ID: @.***>

[andrefdre]

I couldn't find more models in the gazebo database, so I went looking for other alternatives.

I managed to find this tutorial https://www.youtube.com/watch?v=7kEmT-NE75c. With the MakeHumans and blender programs I manage to create people and deploy them in gazebo. Now is just a matter of creating more.

[miguelriemoliveira]

Nice. A lot of people and furniture.

• Miguel Riem de Oliveira Professor Auxiliar * email: @. | @. zoom: https://videoconf-colibri.zoom.us/j/7155402102 office: 22.2.18 | ext: 23887 Dep. de Engenharia Mecânica, Universidade de Aveiro Campus Universitário de Santiago, 3810-193 Portugal

On Mon, Mar 13, 2023 at 4:18 PM André Cardoso @.***> wrote:

I couldn't find more models in the gazebo database, so I went looking for other alternatives.

I managed to find this tutorial https://www.youtube.com/watch?v=7kEmT-NE75c. With the MakeHumans and blender programs I manage to create people and deploy them in gazebo. Now is just a matter of creating more.

[image: image] https://user-images.githubusercontent.com/58526188/224762246-ae58f208-b10d-4167-91e4-beceee830c2d.png

— Reply to this email directly, view it on GitHub https://github.com/DanielCoelho112/synfeal/issues/94#issuecomment-1466459719, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACWTHVXPQSOAVUXNOO7G5WTW35CEDANCNFSM6AAAAAAVMF5NR4 . You are receiving this because you commented.Message ID: @.***>

[andrefdre]

I added more objects and more people. I have a total of 18 people and 12 objects.

Video 4x speedup:

https://user-images.githubusercontent.com/58526188/225718044-13a0292a-cf24-4f95-83a4-fba6385b5a5c.mp4

[pmdjdias]

Pergunta: Verificas se existe colisões entre a camara e os objetos (como se faz com as paredes da sala). É que isso pode levar a ter imagens dentro dos objetos, até se se será sempre uma situação residual.

[miguelriemoliveira]

I like the video. I don't like the camera model. It looks like a big exclamation point. How are you drawing it?

[andrefdre]

I like the video. I don't like the camera model. It looks like a big exclamation point. How are you drawing it?

During the meeting we talked about adding something to the camera, I thought about the exclamation since it was easy to do. I did it, basically adding a cube and a rectangle. The camera model is still bellow it. I will try to draw a camera warning sign to make it look better.

[miguelriemoliveira]

Hi @andrefdre ,

I think the frustum is better. @DanielCoelho112 , do you have an example you can share?

[andrefdre]

I was more thinking maintaining the current kinect camera model and, on top, just adding a camera icon. Just to make it visible.

Something like this:

[miguelriemoliveira]

Hi André,

Better to increase the model size.

But the frustration is even better, I think.

On Fri, Mar 17, 2023, 3:19 PM André Cardoso @.***> wrote:

I was more thinking maintaining the current kinect camera model and, on top, just adding a camera icon. Just to make it visible.

— Reply to this email directly, view it on GitHub https://github.com/DanielCoelho112/synfeal/issues/94#issuecomment-1474001820, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACWTHVRFXXHDQNYNIWE2LOLW4R6G3ANCNFSM6AAAAAAVMF5NR4 . You are receiving this because you commented.Message ID: @.***>

[DanielCoelho112]

Hi @miguelriemoliveira and @andrefdre,

we have a frustum but with marker arrays for the RViz. I don't think it will help here.

https://github.com/DanielCoelho112/synfeal/blob/d472d1c54deaa7f639a807c27d945f6a8792116d/utils_ros.py#L88

I would increase the camera's size and maybe change the color.

[andrefdre]

I would increase the camera's size and maybe change the color.

I tried to change the color but when I add the material property it isn't visible in gazebo and it's visible in rviz. I research about this and it seems like a known problem with gazebo.

[miguelriemoliveira]

How about a bigger size?

On Mon, Mar 20, 2023, 3:08 PM André Cardoso @.***> wrote:

I would increase the camera's size and maybe change the color.

I tried to change the color but when I add the material property it isn't visible in gazebo and it's visible in rviz. I research about this and it seems like a known problem with gazebo.

— Reply to this email directly, view it on GitHub https://github.com/DanielCoelho112/synfeal/issues/94#issuecomment-1476407797, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACWTHVSRHRTWH6RT6MSXRLTW5BXIRANCNFSM6AAAAAAVMF5NR4 . You are receiving this because you were mentioned.Message ID: @.***>

[andrefdre]

How about a bigger size?

This is just about aesthetic, right? If so, I honestly prefer having an icon than increasing the model size. But if I'm the only one, I will try a bigger model.

[miguelriemoliveira]

Hi @andrefdre ,

its not just aesthetic. I mean, these are the images we are going to put in a paper, so they should be good at showing what we want to show.

But I agree that we should not spend a lot more time on this. Just one more suggestion: how about adding to the model of the camera a pyramid representing the view frustum?

@DanielCoelho112 can you post those images from RVIZ, its something like that ...

In any case you should proceed to the training of models.

[andrefdre]

how about adding to the model of the camera a pyramid representing the view frustum?

I think that's possible. Tomorrow I will post a picture of that implementation.

In any case you should proceed to the training of models.

Are the objects enough? How many for test and how many for training?

I also started working today on recreating what the state of the art currently use. I tried two approaches.

First I tried using the same dataset, detect the objects and then place black boxes on them but had many false positives or not detecting some of them.

I then tried to read movement from a dataset with objects and replicate its camera movement. But the black boxes will be random.

Does the teacher have any suggestion?

[miguelriemoliveira]

Are the objects enough? How many for test and how many for training?

The video looks nice. I would try to train with these objects. I would use 80% for training, and 20% for testing.

Does the teacher have any suggestion?

Yes, the first is to create a new issue about this.

Can you do it and then we discuss there.

[andrefdre]

I added a link on top of the camera with a frustum.

[pmdjdias]

Nice! Maybe possible to show as wireframe, but it is a detail anyway.

[DanielCoelho112]

@andrefdre

wireframe would be nice. If possible, something like the ones in RViz (check the architecture from the paper).

[miguelriemoliveira]

Yes, wireframe would be better.

Also, the frustrum should start from the camera's pose, not the icon above.

Finally, the angles of the pyramid could be more realistic. @DanielCoelho112 can give you the correct angles, or you can search in the camera xacro.

[andrefdre]

What does the teacher think @miguelriemoliveira, does it look better?

[miguelriemoliveira]

Much, much better. I was worried for being such a pain, but I am glad I was.

It looks great! Congrats

On Tue, Mar 21, 2023, 20:09 André Cardoso @.***> wrote:

What does the teacher think @miguelriemoliveira https://github.com/miguelriemoliveira, does it look better?

[image: image] https://user-images.githubusercontent.com/58526188/226728984-abf1a10a-225b-4061-8149-098b881fa266.png

— Reply to this email directly, view it on GitHub https://github.com/DanielCoelho112/synfeal/issues/94#issuecomment-1478517784, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACWTHVVVDEHPHBYAR53CT2TW5IDIPANCNFSM6AAAAAAVMF5NR4 . You are receiving this because you were mentioned.Message ID: @.***>

[pmdjdias]

Really nice, yes!

[andrefdre]

Hi,

I finished performing a training with a dataset containing objects. The parameters were:

./rgb_training -fn santuario_95k_obj -mn posenetgooglenet -train_set santuario_95k_obj -test_set santuario_objects_24k -n_epochs 100 -batch_size 45  -loss 'DynamicLoss(sx=0,sq=-1)' -c -im 'PoseNetGoogleNet(True,0.6)' -lr_step_size 150 -lr 1e-4 -lr_gamma 0.5 -wd 1e-3 -gpu 1

The training objects were different from the testing ones and had a total of 22 training objects and 8 testing objects. The loss graph:

It's possible to see that after the 10 epoch, overfitting occurred. Is there something we should do to mitigate this issue? Just one more question for the validation dataset, how big should it be? And it should contain objects, right?

[miguelriemoliveira]

Hi @andrefdre ,

this still does not look good. I think the training is still not going well.

You should run a training dataset without objects (using one of Daniel's datasets) and see if the training actually works. I remember I suggested it before ... did you do it already?

[pmdjdias]

I agree... you shoudl replicate Daniel's work with you dataset to vbe sure the whole pipeline is OK, and then after tray woth object occlusions!

[andrefdre]

You should run a training dataset without objects (using one of Daniel's datasets) and see if the training actually works. I remember I suggested it before ... did you do it already?

Yes, I already did that some weeks ago and it worked perfectly. I think these results are expected due to what we are asking the model to do. I spoke with @DanielCoelho112 and he suggested doing hyperparameter optimization.

[miguelriemoliveira]

Yes, I already did that some weeks ago and it worked perfectly.

Can you please post here the graph of the loss training and test and give some indications on the size of the dataset and other relevant properties in both cases?

I think these results are expected due to what we are asking the model to do.

what do you mean. Can you try to explain better?

I spoke with @DanielCoelho112 and he suggested doing hyperparameter optimization.

Not sure about that ... I mean, it is not easy to find a working combination of hyperparameters. @DanielCoelho112 found one which we know works without objects. Why don't we stick to it? Inserting objects in the scene would not change the task significantly ... it may loose some accuracy, but not stop converging altogether.

I smell a bug ...

[DanielCoelho112]

Hi @andrefdre, @miguelriemoliveira, @pmdjdias,

Can you please post here the graph of the loss training and test and give some indications on the size of the dataset and other relevant properties in both cases?

This was the curve I obtained in the sanctuary with 95k frames:

I believe @andrefdre already obtained similar results, but please share your curves. At first sight, it looks that the model in the previous image is suffering from severe overtiffing. However, that is because of the loss function that is different from training and testing. In training, it is the sum of 2 negative quantities and in test, it is only one.

This explains the difference between the train and test loss.

One thing @andrefdre forgot to mention is that he changed the way we are plotting data. In my curve, as soon as the test loss starts increasing, I stop appending data to the plot. And that is why his test loss keeps increasing. I like this change, in fact, I am doing exactly the same thing in my projects.

In the plot shared by Andre, if we only observe the data until the blue vertical line, it seems close to what I obtained.

Inserting objects in the scene would not change the task significantly ... it may loose some accuracy, but not stop converging altogether.

The plot shared by Andre shows a model that is overfitting, which is different from converging. The model is converging to a minimum, we can see the training loss stabilizing. About the overfitting, we are doing early termination (vertical blue line) and applying some regularization techniques. Since the task changed slightly, I think we should do a few more runs with different hyperparameters to see if we can minimize the overfitting. Possible changes could be: Increasing dropout, increasing weight decay, change Dynamic Loss parameters initiation.

@andrefdre, run the script to produce results on the test data so we can see the magnitude of the errors. It is always good to evaluate the model with objective metrics besides the loss function. Especially when the loss function has learnable parameters.

[miguelriemoliveira]

Ok, so the curve is not so bad.

Now I understand why @DanielCoelho112 suggests tuning the hyper parameters.

Another solution would be to get a dataset with less people in it. Say only one person per frame or something. But I think you should try with this dataset for now.

[andrefdre]

I don't have the results using @DanielCoelho112 datasets anymore, and unfortunately I already removed them from my folder in Deeplar to release space. But I already trained with the same path of the dataset of objects but without the objects and this is the results.

Another solution would be to get a dataset with less people in it. Say only one person per frame or something.

But not always the objects spawns in front of the camera, so only one person would be very sparse.

@andrefdre, run the script to produce results on the test data so we can see the magnitude of the errors. It is always good to evaluate the model with objective metrics besides the loss function. Especially when the loss function has learnable parameters.

As soon as I can, I will do this. Just to get registered what we talked, The validation dataset should be 24k images and use the same objects as the test, since the model never saw them.