search

connormanning / entwine

Entwine - point cloud organization for massive datasets
https://entwine.io
Other
451 stars 128 forks source link

Entwine build fails for LAS file from cloud storage #328

Open fletsch opened 2 days ago

fletsch commented 2 days ago

Hi, first of all thanks for creating this great library!

I recently ran into an issue when building an EPT from a LAS file which is stored on Azure Blob Storage, similiar to issue #294. Running this (entwine version: 3.1.1)

export AZURE_STORAGE_ACCOUNT=some_storage_account_name
export AZURE_STORAGE_ACCESS_KEY=some_acees_key
entwine build -i az://some_container/some_las_file.las -o .

will fail with 

Encountered an error: [json.exception.out_of_range.403] key 'schema' not found
Exiting.

Running the build with the exact same file locally, does work. And interestingly, the issue does not happen for compressed LAZ files.

I tried to find the root of this error and saw that it is caused by this line: https://github.com/connormanning/entwine/blob/2a0524bd74074ed9a1852f51c410ed4d1cf81939/entwine/util/info.cpp#L94 which throws the error

readers.las: Invalid point count: 9917348. Number of points too large for file size.

So it seems the pdal las reader cannot handle the las file with zero points but a header indicating a number of points which was generated here. The corresponding code in pdal is here. However, I could not find out why exactly pdal is validating the file size also for a preview (and especially why for a pointless las files but not for pointless laz files)

A temporary workaround is appending the --deep option for all builds from las files.

Any help is highly appreciated!

connormanning commented 2 days ago

Can you provide the file? Depending on the LAS version, I wonder if it's something like that we are setting the point count field but forgetting about the legacy point count which is getting picked up (or the reverse or something similar). So the file would be helpful to see what's going wrong exactly and to test a fix. Feel free to email it to me if it's not a publicly available file.

fletsch commented 1 day ago

Yes sure, this happend with publicly available files from swissSURFACE3D. I tested it e.g. with this file, but it should be reproducible with any of the files I think.