search

connormanning / entwine

Entwine - point cloud organization for massive datasets
https://entwine.io
Other
451 stars 128 forks source link

Trusted header but not at the start of the file #200

Closed nicolas-chaulet closed 5 years ago

nicolas-chaulet commented 5 years ago

In some file formats the point information are at the end of the file rather than the start (e57 for example). At the moment the only way to read those headers is to set trustedHeaders to false which does not seem ideal (we trust them, we just don't know where they are). I was wondering which format the following line was covering? https://github.com/connormanning/entwine/blob/66a9bf020147e43fe8cdc5e777f776801d2f258a/entwine/builder/scan.cpp#L136

Then we could be explicit and if it is not one of the well known formats (las, laz) then use the add function with trustedHeaders to true. What do you think?

connormanning commented 5 years ago

@nicolas-chaulet can you take a look at #203 (in comparison to your proposed #202). While I agree with your inclination to be more explicit about the types, in lieu of a robust list of file-types where the addRanged feature will work, I think this covers your use-case more generically.

connormanning commented 5 years ago

Note that for some file types (BPF, NITF), they may sometimes work with truncation, but may other times contain trailing metadata and fail, so a simple list of file types cannot necessarily be built to determine whether a truncated scan will work ahead of time.

I think that the bandwidth/speed savings in the case where the data does happen to work correctly make it worth this clumsy workflow of "try truncated, catch errors, try full file if needed".

connormanning commented 5 years ago

Fixed by #203.