MarekKowalski / LiveScan3D

LiveScan3D is a system designed for real time 3D reconstruction using multiple Azure Kinect or Kinect v2 depth sensors simultaneously at real time speed.
MIT License
749 stars 202 forks source link

Azure data corruption, scan merging, color problems, player crashing #54

Closed zachduer closed 3 years ago

zachduer commented 3 years ago

Hello! First, of all, thank you so much for the work you have done on this, Marek! It was really fun to find this tool and get a chance to play with it. I've run into several issues that I'd like to note, two of which have already been posted in previous issues but seemingly are still an issue. I'm not sure if you're interested in still actively working on this software, but let me know!

1) The only method for recording that has worked for me is ASCII ply with merged scans enabled. If I use binary ply and/or disable merge scans, then it fails to save all of the frames, and the frames it does save are badly corrupted. Many of the points are missing and their colors have turned to noise. Interestingly, when I save with ASCII ply and merge scans enabled, and then view the recording in the player and enable the save frames option, it appears to correctly export them as binary ply to the outplayer folder without corruption.

2) When I do record with ASCII ply, I'm able to open the saved frames in both the LiveScan3DPlayer and MeshLab, to check them (although, Meshlab is angry about an empty line in the header of the ASCII file, which can be easily deleted). However, even though I have "merge scans" enabled, it's clearly only actually saving the points from one of the 3 cameras I'm using. I can't adequately test what happens when I disable "merge scans," as noted above. This appear to be the same as Issue #41, if I'm understanding what they said correctly.

3) As reported in Issue #47, there's a color problem (at least with saving as ASCII ply and merge scans). Just like they're saying, the ply files and bin files seem to have a color channel problem, with the vertex colors appearing very blue. I can fix this in individual files by going into the ASCII ply and reordering the properties in the header as red, blue, green instead of the default red, green, blue. So, seems like a simple fix that I could do in the source or in post.

4) The LiveScan3DPlayer always crashes after playing. I have to close it from the task manager. I haven't done any work to track down exactly why, but it has happened so far in every circumstance that I've ever run it.

For me, fixing the second issue of the scan merging would be enough for me to be able to use this software for my project, so I'd love to know if you have any thoughts! Perhaps I'm doing something wrong or misunderstanding, which would be the best possible scenario!

Again, thanks very much, Zach

ChristopherRemde commented 3 years ago

Hi Zachduer,

I also noticed the artifacts that you've been describing. A lot of the problems you have described have already been fixed, but are not yet on the Azure Kinect branch. I could offer to do a PR with fixes, when I have some time (Could take 1-2 weeks).

1: Are you using multiple Azure Kinects on one PC? Because I noticed corrupted files are happening when more than one Kinect is connected to a PC. The cause of this is that all Kinects write to the same temporary file on the client device before transmitting, which is corrupting the data. I already fixed this in PR #49 , but I'm now realizing I should have done a seperate PR for this.

2: I haven't experienced this one yet, but I can do a little investigation.

3: This should be an easy fix, I'll try to replicate this issue

Additionally to the issues you mentioned, there is the PR #38 which fixes problems with the PLY format. This has been fixed on the Default Kinect Branch, but not on the Azure Kinect branch yet.

I'll try to commit a PR soon which should fix most of the problems.

Regarding 4: This also happens for me, it always crashes and is not in a usable state. However I don't know if I'll have the time to figure this one out.

zachduer commented 3 years ago

Hi Chris! Thank you for your response! Yes, I am using all 3 Azure Kinects on a single PC. I will take a look at PR #49 and work from there, thank you so much!!

ChristopherRemde commented 3 years ago

Hi Zach,

yeah that should probably be the cause for most of your issues!

Let me know if #49 fixes the problems for you! The pull request adds temporal synchronisation to LiveScan, but you can just leave it disabled.

zachduer commented 3 years ago

Hi Chris!

Yes, confirmed, PR #49 fixes all of the issues I mentioned here. Thank you so much! The hard work of you all has saved me SO much time, so I get to skip to the fun parts! <3