ConnectionResetError: [Errno 104] Connection reset by peer

[lluisb3]

Hi @jdibenes and thanks a lot for the contribution. I am working in a MR project and we are trying to test the data extracted using the HoloLens 2. The main idea is to obtain the pointcloud of a full scene to later on train an algorithm. Now we are in the testing phase but later on it will ideal even to create our own dataset using the HoloLens 2 headset and potentially this library. The problem that I cannot achieve the transfer of data between the Hololens and my pc. I am usinf the following specifications:

pc OS: Linux 20.04

Miniconda env with: Python 3.9 I have install the libraries opencv-python, av, numpy, pynout and open3d.

I have achived the Preparation and Installation steps. (installed the version v1.0.25.0) I have checked the connectivity of the headset and my pc accesing the data portal via HoloLens IP address.

Then I am trying to run the python script client_stream_pv.py and I got the following error:

Traceback (most recent call last):
  File "/home/ither1/hl2ss/viewer/client_stream_pv.py", line 83, in <module>
    client.open()
  File "/home/ither1/hl2ss/viewer/hl2ss.py", line 1395, in open
    self.get_next_packet()
  File "/home/ither1/hl2ss/viewer/hl2ss.py", line 1400, in get_next_packet
    data = super().get_next_packet()
  File "/home/ither1/hl2ss/viewer/hl2ss.py", line 741, in get_next_packet
    return self._client.get_next_packet()
  File "/home/ither1/hl2ss/viewer/hl2ss.py", line 388, in get_next_packet
    self._unpacker.extend(self._client.recv(self._chunk_size))
  File "/home/ither1/hl2ss/viewer/hl2ss.py", line 276, in recv
    chunk = self._socket.recv(chunk_size)
ConnectionResetError: [Errno 104] Connection reset by peer

Any idea about what could be happening? Thanks in advance!

[srafaelm]

I am also getting the same problem at first I thought it was the resolution of the stream but I am using one that is supported so I don't know.

[jdibenes]

Hello, Are you able to stream data from other sensors? For example, if you run client_stream_rm_vlc.py do you get any video? What is your HoloLens OS version?

[lluisb3]

I did not do nothing, I try to run everything again to check if working to answer you back with the problems but now everything is working. Sorry for the disturbance.

But now that everything is working, there is a way to record directly the pointcloud in a .ply format? I am trying the simple_recorder.py function and saves the outputs:

extended_audio.bin extended_audio.csv personal_video.bin personal_video.csv video.mp4 .. if chossen also gyroscope, accelerometer, etc

But everything is saved in .bin and .csv formats. The idea is to record the outputs as shown, for example, in the function sample_rgbd.py, the pointcloud in .ply format. Thanks a lot in advance! Impressive job.

HoloLens OS version: Windows Holographic for Business OS build: 22621.1244

[jdibenes]

There is no support for recording to .ply directly. A possible approach would be to record depth and color data to .bin using simple_recorder.py, then create depth-color image pairs from the .bin files similarly to simple_sequencer.py, then create the pointclouds like in sample_rgbd_pv.py and save them as .ply using Open3D.

[lluisb3]

Thanks a lot for the answer and for this repository again! I am gonna work in this following your suggestions.

[lluisb3]

Hi @jdibenes, I followed your suggestions and I have already obtain the pointclouds in .ply file for each frame. I reopen the issue to suggest you one thing and to ask another.

The suggestion is that I think you are missing this when streaming in sample_rgbd_pv.py the pointclouds, correct my if I am wrong:

pcd.transform([[1,0,0,0],[0,-1,0,0],[0,0,-1,0],[0,0,0,1]])

If not the pointcloud streamed is upside-down, at least in my case.

Then I wanted to ask if you already know the correct way to combine these point clouds to obtain a single point cloud of the scene. I try first to combine directly all the point clouds with pcd_combined += down suugested in open3d library but the result is this:

With all pointclouds in the same space. Then I tried to register first the point clouds following this tutorial:

http://www.open3d.org/docs/latest/tutorial/Advanced/multiway_registration.html

But the registration is slow and not very accurated as shown:

I was wondering if you already experiebces this problem and know the solution maybe taking advance of the spatial information recorded from the HoloLens or another possible approach.

Thanks in advance!

[jdibenes]

Try using the camera extrinsics and pose like in sample_integrator_pv.py.

[lluisb3]

That is it. It worked! Thanks a lot, you have done an amazing job in this project.

[lluisb3]

Hi @jdibenes! I reopen this issue with a small question. I followed your suggestion and I can obtain the pointcloud of a scene directly from Hololens2, but do you know if is there a way to extract directly the 3D mesh surface reconstruction from Hololens2 without any reconstruction algorithm?

Thanks in advance!

[jdibenes]

client_ipc_sm.py downloads the spatial mapping mesh stored in the HoloLens.

[lluisb3]

Thanks @jdibenes for the clarification, that is exactly what I was looking for. The only thing now is that this spatial mapping mesh downloaded from Hololens is not very a high quality I think becasue is performed only once, without updating while examinating the room. I was wondering if there is a way to improve the mesh by refining triangles and vertex on the fly as you do to obtain the final pointcloud on sample_integrator_pv.py where pointclouds are combined to extract a final pointcloud of the room.

Also the colors of the mesh are given depending on the direction of the normals and no the real colors of the image as you can see in the image below. I guess maybe is also possible to update the colors while examinating the room taking advantage of the rgb image coming from the pv_camera.

Thanks a lot in advance!

[jdibenes]

Try increasing the triangles per cubic meter (tpcm). It may be possible to refine the meshes but I don't know. Maybe it is possible to colorize the mesh by projecting the vertices onto the pv image but it might not look good because there are fewer points, the integrator pv sm script shows how to align the open3d point cloud and the spatial mapping mesh.

[lluisb3]

Okay. I will follow your suggestions.

Thanks a lot for all the help @jdibenes and again thanks for the all the time you put in this repository, it is a real nice tool!