Indexing Crashes - PotreeConverter 2.0

[smcavoy12]

Potree 2.0 seems to failing a lot processing different datasets. No error is returned, the process just stops and exits. I'm changing my original post, i thought it had to do with structured data grids, but it'd having trouble on lots of other samples I'm running, regardless of format.

I've attached a test cube in laz format, 300x300x300 meters with each point spaced exactly 1 meter apart. cube.zip

Below is a version processed with 1.7 https://pointcloud.ucsd.edu/cube.html

also attaching log. gets to 98% indexing. cube_log.txt

[lcrumbliss]

I too am experiencing a sudden program exit during the indexing phase of a large .las file (22Gb, about 1.1 Billion points). I am trying converter v2.0 because v1.7 is currently processing a copy of this same file and is calculated to take weeks to process the same las file. I've attached a log of v2.0's output. Note the indexing section. And, hm, that boundary cube. That seems suspicious.

By the way, I don't recall v1.7 of the converter taking THIS long in the past on large las files. Files half that size finished in hours If I recall correctly. But in this case, it's taking about an hour per 1 million points. Does this, along with the issue of v2.0 exiting suddenly point to a problem with the las file?

log.txt

[m-schuetz]

Thanks for the reports, I'll take a look at it.

@lcrumbliss
Would it be possible for you to also send me a laz file to mschuetz@potree.org? The smaller the laz that causes the issue, the better.

[m-schuetz]

@smcavoy12

Problem is that the converter wasn't made with volume data in mind. The crash is caused because the subsampling array has a fixed size of 1 million points and it is populated by access via an index (=number of already accepted points) which causes an out of range error here. I'm currently testing a fix to the crashes, but it will nevertheless perform rather poorly if the space is completely filled with points. May I ask, what is your use case? Even if the structure is built, it's not particularly suitable for rendering volume data.

[m-schuetz]

@smcavoy12 Addressing the mentioned issue fixes the crash, but now the converter runs forever. Probably because a volume completely filled with points increases subsampling duration exponentially compared to a surfe within a volume. I'll have to take a deeper look into that issue.

@lcrumbliss Your issue might be a different one so if you can, send me a test data set that I can use to replicate and debug the issue.

[lcrumbliss]

@m-schuetz I will convert the las file to an laz file in LASTools. Then I'll email you some log-in credentials to our extranet where you can download the file from. Does that sound OK?

[m-schuetz]

Sounds good!

[lcrumbliss]

@m-schuetz OK. Will take a while to convert to a laz file. I will email you once everything is ready.

[lcrumbliss]

@m-schuetz OK, account created and email with instructions sent.

[m-schuetz]

thanks, I'll be looking at it as soon as possible

[m-schuetz]

@lcrumbliss There are a couple of issues here, one of which is that the bounding box is 1000x larger than it should be. It looks like almost the entire data set is within a box of size [400, 400, 177], but according to the las header the bounding box has a size of roughly [783370, 130523, 400]. Some subregions are able to be converted, like min: [1, 1, -2], max: [2, 2, 2], but the converter still crashes if the region covers the 400/400/177 size bounding box.

I'll do some futher debugging in the next days. However, the bounding box has to be fixed because the converter as well as the renderer rely on meaningful bounding boxes. If the bounding box is too large by a factor 1000x, it would essentially lead to 10 virtually empty octree levels.

[lcrumbliss]

@m-schuetz Thanks Markus. Yeah, I was coming to suspect that bounding cube. Seriously, it was like 150 miles x 150 miles. Thanks for taking a look.

[m-schuetz]

@lcrumbliss

I've taken another look and found the second main issue, which is that there appear to be around 600 million points with coordinate 0/0/0. You can use following lastools command to repair the las file into something that is convertable:

las2las -i final_las.laz -drop_xyz -0.01 -0.01 -0.01 0.01 0.01 0.01 -o without_0.las
las2las -i without_0.las -keep_xyz -400.0 -400.0 -400.0 400.0 400.0 400.0 -o convertable.las

Still looks like the file contains multiple non-aligned single scan positions that have to be aligned before conversion.

[m-schuetz]

@smcavoy12 Taking a closer look at the cube again, just wanted to note that the compression of the cube is crazy:

The laz is 1400 times smaller than the las file ^^'

[lcrumbliss]

@lcrumbliss

I've taken another look and found the second main issue, which is that there appear to be around 600 million points with coordinate 0/0/0. You can use following lastools command to repair the las file into something that is convertable:

las2las -i final_las.laz -drop_xyz -0.01 -0.01 -0.01 0.01 0.01 0.01 -o without_0.las
las2las -i without_0.las -keep_xyz -400.0 -400.0 -400.0 400.0 400.0 400.0 -o convertable.las

Still looks like the file contains multiple non-aligned single scan positions that have to be aligned before conversion.

Thanks @m-schuetz. It's becoming more and more obvious how badly this laz file was formatted. I appreciate you investigating.

[m-schuetz]

It's fine, I'll need to make PotreeConverter more robust and/or verbose towards failure cases so it's good to know what to expect.

[smcavoy12]

@m-schuetz That's crazy! I hadn't noticed, thanks for pointing it out. I'm going to have to share it with my colleagues for sure, we'll want to point to this pure example in our paper.

I'm trying to get clearance to share our use case, I think you'll find it compelling. Thank you for your consideration! I may have to hit you up via email.

[m-schuetz]

I may have to hit you up via email.

Please do! Since this is now under a partial commercial license, I'm kinda obligated to make it as robust as possible.

[m-schuetz]

@smcavoy12

Hi, can you try this hotfix here? PotreeConverter_2.0_windows_x64_hotfix_bug_427.zip

However, you'll have to try with the "random" sampling strategy because the poisson-disk sampling strategy is entirely unsuitable for this data set. The random sampling strategy creates the octree in around 7 seconds, but it has some sampling patterns. PotreeConverter.exe cube.laz -o ./cube_converted -m random

The root node already has around 2 million points and since there is no compression, it takes 70MB which will be quite slow to load. Compression in PotreeConverter 2.0 is scheduled for around end of september.

[m-schuetz]

Relevant commit that fixes the cube sample when -m random is used: https://github.com/potree/PotreeConverter/commit/097c1d4749fed11bb048c3469f9edeb338bf4c23

The default poisson disk strategy still fails to finish, presumably because there are way too many distance checks since the space is completely filled.

[smcavoy12]

@m-schuetz

Yes, it works for me, thanks for popping out a fix so quickly! And I've tried it on a few different larger datasets too, I definitely prefer the poisson cluster pattern, but on most of my datasets it's not anywhere near as noticeable as it is here.

https://pointcloud.ucsd.edu/cube2.html

I am open to doing mulitple pointclouds in slices if that's required, or splitting the cloud in two with some kind of random sampling. Would that be the same problem if the boundaries are the same but the infill is different?

Thanks so much!

[dpfaff99]

I'm having a similar issue with a point cloud that was created in Register360 and sub-sampled in Cloud Compare. The process stops early with no error. I've tried additional re-sampling and using Lastools for conversion. I have completed other files using 2.0 and the same workflow with no problem. I tried the hotfix from above and had a similar result. Below is link to a small version of the point cloud (10 million) points and the text from the cmd window as well as the output from potree: https://wlu.box.com/s/gg25g9gonyonugavgfxgujnxvfanqkxq

Here is a previous 2.0 conversion https://3d.wlu.edu/v20/duomo_only/duomo99.html

[m-schuetz]

@dpfaff99

Sorry, I missed that, checking it right now.

[m-schuetz]

@dpfaff99

Your file contains a large number of duplicate points. This will mess with the converter because the converter partitions the octree based on the number of points in a node, but if all the points have the same coordinate, the converter keeps on partitioning endlessly until it crashes.

I've comitted a "fix" that prints a warning and aborts further recursive partitioning, if a partition is just as large as the input: https://github.com/potree/PotreeConverter/commit/71cc41300d776b4cdeeba6dcbef198d9f0826fb7

This isn't a proper fix, however, and more of a hint that the las file needs to be repaired.

e.g.:

las2las.exe -i Bargello10Mtest.las -o duplicates.las -keep_xyz -3.1630329834204787 0.15911754937479827 -0.16870976629438239 -3.1628692882388805 0.15928124455639647 -0.16854607111278419

will produce a file that contains the 12879 identical coordinates within the specified bounding box.

-3.163012966617998 0.159183058295994 -0.1686877503108
-3.163012966617998 0.159183058295994 -0.1686877503108
-3.163012966617998 0.159183058295994 -0.1686877503108
-3.163012966617998 0.159183058295994 -0.1686877503108
-3.163012966617998 0.159183058295994 -0.1686877503108
-3.163012966617998 0.159183058295994 -0.1686877503108
-3.163012966617998 0.159183058295994 -0.1686877503108
...

[m-schuetz]

Actually, I think I'm going to change this so that Potree aborts with an error rather than ignoring the issue and printing a warning.

[m-schuetz]

PotreeConverter now throws an error message like this:

ERROR(C:\dev\workspaces\PotreeConverter\2.0\Converter\src\indexer.cpp:904): a non-partitionable sequence of points was encountered, which may be caused by a large number of duplicates. #points in box: 12879, #unique points in box: 1, min: -3.16762, 0.155353, -0.172638, max: -3.16238, 0.160591, -0.1674