search

r-lidar / rlas

R package to read and write las and laz files used to store LiDAR data
https://cran.r-project.org/package=rlas
GNU General Public License v3.0
34 stars 14 forks source link

Invalid reading of lax file on windows #50

Closed Jean-Romain closed 3 years ago

Jean-Romain commented 3 years ago

Based on https://github.com/Jean-Romain/lidR/issues/408 and using the las + lax files reported there. This issue is only reproducible on Windows

Expected behaviour (tested on R 4.0.3, Linux, rlas 1.3.9)

las <- rlas::read.las('issue 408/157668.5_243478.3.las', filter = "-inside 157980 243480 158270 243770")
dim(las)
#> [1] 8143683      18

Observed behaviour (tested on R 4.0.3, Windows, rlas 1.3.9)

las <- rlas::read.las('issue 408/157668.5_243478.3.las', filter = "-inside 157980 243480 158270 243770")
dim(las)
#> [1] 30595    18

Oberservation:

One last thing to test on windows with LAStools (tested on Linux)

las2las -i 157668.5_243478.3.las -inside 157980 243480 158270 243770 -o clip.las
lasinfo clip.las
> extended number of point records: 8143683
adrienmichez commented 3 years ago

Here is the result of lasinfo:

lasinfo -i "C:\TMP\lidR_bug\transfer_85074_files_52cf1058\clip.las"
>  extended number of point records: 8143683
Jean-Romain commented 3 years ago

I tried with rlas 1.3.7, 1.3.9 and 1.4.0 either compiling in C++11 and C++14. They all failed to perform the query on Windows. So the bug actually exist for a long time. I can also reproduce with first returns only:

las2las -first_only -i 157668.5_243478.3.las -o test.las
lasindex -i test.las

Linux

las <- rlas::read.las('~/issue 408/test.las', filter = "-inside 157980 243480 158270 243770")
dim(las)
#> [1] 5292612      18

Windows

las <- rlas::read.las('~/issue 408/test.las', filter = "-inside 157980 243480 158270 243770")
dim(las)
#> [1] 686183     18

Rplot

adrienmichez commented 3 years ago

Does that mean that we should not use spatial indexing in windows?? This has quite broad implications!

Jean-Romain commented 3 years ago

Have you ever encountered other issues on windows ? You are the first one to report an problem and so far we were only able to reproduce on this file. It is definitively a complex problem that needs much more investigation.

adrienmichez commented 3 years ago

Sure... But I don't want to manually check if something went wrong after using spatial indexing. I hope you will soon find how to fix this :-)

Jean-Romain commented 3 years ago

Bad new for today. I've updated rlas with the latest source code of LASlib and LASzip and it is still failing :disappointed:

Jean-Romain commented 3 years ago

I made very little progress today. It seems that the coordinates of some points are badly read for unkown reasons. This starts at interval [72200247, 72517435]

Linux

> las <- rlas::read.las('~/Téléchargements/issue 50/157668.5_243478.3.las', filter = "-inside 157980 243480 158270 243770")
16 cells of 156900/242600 158500/244200 intersect rect 157980/243480 158270/243770
LASindex: used 16 cells of total 182
Interval start = 2411200, end = 2678575, full = 15060790, total = 16622934, cells = 16 
   Point (157933.73, 243496.61) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157980.84, 243478.93) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 5084464, end = 5285031, full = 15060790, total = 16622934, cells = 16 
   Point (157929.04, 243443.84) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157958.77, 243464.21) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 7596484, end = 7819087, full = 15060790, total = 16622934, cells = 16 
   Point (157915.35, 243425.31) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157971.41, 243410.94) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 9948801, end = 9968545, full = 15060790, total = 16622934, cells = 16 
Interval start = 9975888, end = 10007926, full = 15060790, total = 16622934, cells = 16 
   Point (157902.24, 243378.48) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 10014607, end = 10045098, full = 15060790, total = 16622934, cells = 16 
Interval start = 10051854, end = 10081465, full = 15060790, total = 16622934, cells = 16 
Interval start = 10089090, end = 10118304, full = 15060790, total = 16622934, cells = 16 
   Point (157970.58, 243399.31) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 10126859, end = 10151486, full = 15060790, total = 16622934, cells = 16 
Interval start = 10161245, end = 10191604, full = 15060790, total = 16622934, cells = 16 
Interval start = 72200247, end = 72517435, full = 15060790, total = 16622934, cells = 16 
   Point (157915.09, 243775.11) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157950.81, 243804.03) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157977.76, 243773.96) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 74745515, end = 75102684, full = 15060790, total = 16622934, cells = 16 
   Point (157890.74, 243766.12) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157946.23, 243765.06) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157963.39, 243742.14) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 77246009, end = 77609879, full = 15060790, total = 16622934, cells = 16 
   Point (157910.84, 243733.35) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157926.59, 243732.23) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157963.89, 243737.67) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 80070354, end = 80392642, full = 15060790, total = 16622934, cells = 16 
   Point (157900.10, 243693.55) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157944.15, 243688.27) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157961.20, 243692.20) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 83094087, end = 83456284, full = 15060790, total = 16622934, cells = 16 
   Point (157902.93, 243668.63) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157931.57, 243666.55) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157969.85, 243664.62) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 86199040, end = 86575969, full = 15060790, total = 16622934, cells = 16 
   Point (157903.58, 243616.46) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157925.78, 243621.48) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157968.21, 243635.94) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 89298031, end = 89650155, full = 15060790, total = 16622934, cells = 16 
   Point (157907.23, 243580.75) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157932.89, 243609.17) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157970.16, 243581.41) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 92151662, end = 92501280, full = 15060790, total = 16622934, cells = 16 
   Point (157893.44, 243554.00) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157939.32, 243559.21) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157974.29, 243567.49) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 95015614, end = 95316926, full = 15060790, total = 16622934, cells = 16 
   Point (157912.87, 243520.20) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157963.63, 243530.99) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 96375793, end = 97411041, full = 15060790, total = 16622934, cells = 16 
   Point (157944.13, 243512.19) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158043.70, 243473.14) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158137.91, 243469.89) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158233.12, 243478.33) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158283.49, 243481.59) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 97475634, end = 98317548, full = 15060790, total = 16622934, cells = 16 
   Point (158020.57, 243445.61) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158057.78, 243446.28) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158074.59, 243457.57) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158147.17, 243442.03) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158179.26, 243457.68) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158203.35, 243442.32) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158215.34, 243460.07) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158254.61, 243464.20) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158292.17, 243452.65) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 98391434, end = 99222479, full = 15060790, total = 16622934, cells = 16 
   Point (158058.71, 243429.41) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158069.05, 243410.91) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158093.19, 243415.99) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158124.65, 243420.24) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158158.73, 243438.15) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158234.87, 243427.04) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158265.06, 243427.81) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158283.69, 243433.78) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99279624, end = 99298414, full = 15060790, total = 16622934, cells = 16 
Interval start = 99306363, end = 99378500, full = 15060790, total = 16622934, cells = 16 
   Point (158001.77, 243384.69) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99387501, end = 99426476, full = 15060790, total = 16622934, cells = 16 
Interval start = 99432657, end = 99474449, full = 15060790, total = 16622934, cells = 16 
   Point (158085.84, 243400.46) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99482509, end = 99560218, full = 15060790, total = 16622934, cells = 16 
Interval start = 99566800, end = 99629829, full = 15060790, total = 16622934, cells = 16 
   Point (158116.74, 243399.11) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99637222, end = 99693219, full = 15060790, total = 16622934, cells = 16 
   Point (158171.97, 243396.99) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99699866, end = 99756676, full = 15060790, total = 16622934, cells = 16 
Interval start = 99764528, end = 99836639, full = 15060790, total = 16622934, cells = 16 
   Point (158207.29, 243397.96) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99845578, end = 99922488, full = 15060790, total = 16622934, cells = 16 
   Point (158263.93, 243385.47) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99931411, end = 100003373, full = 15060790, total = 16622934, cells = 16 
Interval start = 100012971, end = 100055900, full = 15060790, total = 16622934, cells = 16 
   Point (158283.96, 243383.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 118993732, end = 119007854, full = 15060790, total = 16622934, cells = 16 
Interval start = 119017700, end = 119063855, full = 15060790, total = 16622934, cells = 16 
Interval start = 119070113, end = 119071244, full = 15060790, total = 16622934, cells = 16 
Interval start = 119082891, end = 119138922, full = 15060790, total = 16622934, cells = 16 
   Point (158052.16, 243807.77) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119145653, end = 119193799, full = 15060790, total = 16622934, cells = 16 
Interval start = 119202638, end = 119204102, full = 15060790, total = 16622934, cells = 16 
Interval start = 119219563, end = 119277431, full = 15060790, total = 16622934, cells = 16 
   Point (158100.80, 243806.44) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119285670, end = 119287674, full = 15060790, total = 16622934, cells = 16 
Interval start = 119295790, end = 119295847, full = 15060790, total = 16622934, cells = 16 
Interval start = 119301997, end = 119363882, full = 15060790, total = 16622934, cells = 16 
   Point (158120.94, 243804.61) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119372969, end = 119374288, full = 15060790, total = 16622934, cells = 16 
Interval start = 119388109, end = 119447150, full = 15060790, total = 16622934, cells = 16 
Interval start = 119455764, end = 119457245, full = 15060790, total = 16622934, cells = 16 
Interval start = 119465232, end = 119465318, full = 15060790, total = 16622934, cells = 16 
Interval start = 119471656, end = 119526017, full = 15060790, total = 16622934, cells = 16 
   Point (158183.29, 243807.22) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119533834, end = 119535365, full = 15060790, total = 16622934, cells = 16 
Interval start = 119549555, end = 119597124, full = 15060790, total = 16622934, cells = 16 
Interval start = 119605842, end = 119607453, full = 15060790, total = 16622934, cells = 16 
Interval start = 119623613, end = 119689314, full = 15060790, total = 16622934, cells = 16 
   Point (158266.54, 243807.02) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119698684, end = 119738796, full = 15060790, total = 16622934, cells = 16 
Interval start = 119811561, end = 120617070, full = 15060790, total = 16622934, cells = 16 
   Point (158009.29, 243785.15) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158048.24, 243787.49) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158114.88, 243795.65) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158129.80, 243790.60) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158153.09, 243774.73) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158223.63, 243782.31) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158264.93, 243786.68) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (158270.94, 243794.14) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 120699724, end = 121585718, full = 15060790, total = 16622934, cells = 16 
Interval start = 121659673, end = 122588799, full = 15060790, total = 16622934, cells = 16 
   Point (158284.51, 243732.69) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 122671044, end = 123647049, full = 15060790, total = 16622934, cells = 16 
   Point (158293.52, 243676.78) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 123741675, end = 124603093, full = 15060790, total = 16622934, cells = 16 
   Point (158271.68, 243675.86) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 124682798, end = 125518909, full = 15060790, total = 16622934, cells = 16 
   Point (158281.31, 243617.25) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 125594171, end = 126463883, full = 15060790, total = 16622934, cells = 16 
   Point (158300.39, 243591.28) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 126582491, end = 127436464, full = 15060790, total = 16622934, cells = 16 
   Point (158282.86, 243557.06) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 127553895, end = 128442590, full = 15060790, total = 16622934, cells = 16 
   Point (158276.12, 243527.84) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 128551794, end = 129319764, full = 15060790, total = 16622934, cells = 16 
   Point (158271.99, 243516.16) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Read 16622934 points
Read 8143683 points in bbox

Windows

las <- rlas::read.las('issue 50/157668.5_243478.3.las', filter = "-inside 157980 243480 158270 243770")
16 cells of 156900/242600 158500/244200 intersect rect 157980/243480 158270/243770
LASindex: used 16 cells of total 182
Interval start = 2411200, end = 2678575, full = 15060790, total = 16622934, cells = 16 
   Point (157933.73, 243496.61) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157980.84, 243478.93) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 5084464, end = 5285031, full = 15060790, total = 16622934, cells = 16 
   Point (157929.04, 243443.84) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157958.77, 243464.21) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 7596484, end = 7819087, full = 15060790, total = 16622934, cells = 16 
   Point (157915.35, 243425.31) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (157971.41, 243410.94) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 9948801, end = 9968545, full = 15060790, total = 16622934, cells = 16 
Interval start = 9975888, end = 10007926, full = 15060790, total = 16622934, cells = 16 
   Point (157902.24, 243378.48) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 10014607, end = 10045098, full = 15060790, total = 16622934, cells = 16 
Interval start = 10051854, end = 10081465, full = 15060790, total = 16622934, cells = 16 
Interval start = 10089090, end = 10118304, full = 15060790, total = 16622934, cells = 16 
   Point (157970.58, 243399.31) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 10126859, end = 10151486, full = 15060790, total = 16622934, cells = 16 
Interval start = 10161245, end = 10191604, full = 15060790, total = 16622934, cells = 16 
Interval start = 72200247, end = 72517435, full = 15060790, total = 16622934, cells = 16 
   Point (-2380267.51, 6309209.15) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-1201272.31, 7399820.30) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-2325872.62, -5888351.17) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 74745515, end = 75102684, full = 15060790, total = 16622934, cells = 16 
   Point (168468.82, 9866426.47) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (655.53, 3829931.99) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (655.53, 21646.23) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (655.70, 9866426.47) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 77246009, end = 77609879, full = 15060790, total = 16622934, cells = 16 
   Point (4227240.09, 14629603.68) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (6127128.73, 13868075.36) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (8919617.69, 15140129.12) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 80070354, end = 80392642, full = 15060790, total = 16622934, cells = 16 
   Point (518389.77, -17634368.44) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (1372979.21, -5286788.79) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-1310717.43, 4624370.70) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (492833.29, -540613.56) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 83094087, end = 83456284, full = 15060790, total = 16622934, cells = 16 
   Point (168468.81, 21647.50) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (655.86, 9866426.47) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (655.87, 3830604.08) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 86199040, end = 86575969, full = 15060790, total = 16622934, cells = 16 
   Point (15029373.28, -17507283.33) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (15360985.44, -19331805.57) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (14382532.96, -19356053.89) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (13392284.00, -17684885.89) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 89298031, end = 89650155, full = 15060790, total = 16622934, cells = 16 
   Point (-35876.71, 15251540.32) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (3604648.09, 13423741.29) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-15603953.51, 11889543.51) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-11607568.22, 12917147.99) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 92151662, end = 92501280, full = 15060790, total = 16622934, cells = 16 
   Point (33586.30, 643563.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11549.12, -1310719.99) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (13433464.84, 593758.73) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 95015614, end = 95316926, full = 15060790, total = 16622934, cells = 16 
   Point (11016089.07, 157262.77) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016980.28, 157340.85) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016069.01, 157375.08) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 96375793, end = 97411041, full = 15060790, total = 16622934, cells = 16 
   Point (-2560491.51, -6112484.29) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (1160645.13, -19362610.87) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-2599813.11, -16791575.49) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (1587281.93, -6605314.99) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (1207173.13, -17684889.27) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-1420162.55, -2887365.62) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (1492910.09, 13322871.89) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (587205.13, -20036262.84) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (1150812.17, 10632560.97) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (1502087.69, -18508618.67) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 97475634, end = 98317548, full = 15060790, total = 16622934, cells = 16 
   Point (2695956.07, 1056442.89) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (13443874.04, 1429342.73) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11568.80, -1310719.99) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (22063.44, -1528954.87) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11595.84, 1355939.85) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11734.12, -2654207.99) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (22029.36, -1528954.87) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (33821.57, 1350696.97) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (22447.99, -2641100.79) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 98391434, end = 99222479, full = 15060790, total = 16622934, cells = 16 
   Point (20934819.84, 655.69) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-4086824.96, 168632.49) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-14041743.36, 655.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (6913392.64, 655.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (13746831.36, 655.70) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-16145448.96, 655.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (-14488698.88, 860.33) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (20101201.92, 168468.82) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99279624, end = 99298414, full = 15060790, total = 16622934, cells = 16 
Interval start = 99306363, end = 99378500, full = 15060790, total = 16622934, cells = 16 
   Point (758909.45, -12392143.80) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99387501, end = 99426476, full = 15060790, total = 16622934, cells = 16 
Interval start = 99432657, end = 99474449, full = 15060790, total = 16622934, cells = 16 
   Point (11712.88, 1396572.18) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99482509, end = 99560218, full = 15060790, total = 16622934, cells = 16 
   Point (9199946.08, -14001107.33) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99566800, end = 99629829, full = 15060790, total = 16622934, cells = 16 
Interval start = 99637222, end = 99693219, full = 15060790, total = 16622934, cells = 16 
   Point (-21318205.44, 655.70) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99699866, end = 99756676, full = 15060790, total = 16622934, cells = 16 
Interval start = 99764528, end = 99836639, full = 15060790, total = 16622934, cells = 16 
   Point (-16852578.69, 18006671.36) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99845578, end = 99922488, full = 15060790, total = 16622934, cells = 16 
   Point (2695881.76, -3122132.47) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 99931411, end = 100003373, full = 15060790, total = 16622934, cells = 16 
   Point (3830599.85, 14088782.95) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 100012971, end = 100055900, full = 15060790, total = 16622934, cells = 16 
Interval start = 118993732, end = 119007854, full = 15060790, total = 16622934, cells = 16 
Interval start = 119017700, end = 119063855, full = 15060790, total = 16622934, cells = 16 
   Point (-15929584.93, 11016089.07) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119070113, end = 119071244, full = 15060790, total = 16622934, cells = 16 
Interval start = 119082891, end = 119138922, full = 15060790, total = 16622934, cells = 16 
Interval start = 119145653, end = 119193799, full = 15060790, total = 16622934, cells = 16 
   Point (9866418.13, -4865150.47) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119202638, end = 119204102, full = 15060790, total = 16622934, cells = 16 
Interval start = 119219563, end = 119277431, full = 15060790, total = 16622934, cells = 16 
Interval start = 119285670, end = 119287674, full = 15060790, total = 16622934, cells = 16 
Interval start = 119295790, end = 119295847, full = 15060790, total = 16622934, cells = 16 
Interval start = 119301997, end = 119363882, full = 15060790, total = 16622934, cells = 16 
   Point (-10842768.67, 11016089.07) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119372969, end = 119374288, full = 15060790, total = 16622934, cells = 16 
Interval start = 119388109, end = 119447150, full = 15060790, total = 16622934, cells = 16 
   Point (-11466665.82, 8976468.31) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119455764, end = 119457245, full = 15060790, total = 16622934, cells = 16 
Interval start = 119465232, end = 119465318, full = 15060790, total = 16622934, cells = 16 
Interval start = 119471656, end = 119526017, full = 15060790, total = 16622934, cells = 16 
Interval start = 119533834, end = 119535365, full = 15060790, total = 16622934, cells = 16 
Interval start = 119549555, end = 119597124, full = 15060790, total = 16622934, cells = 16 
   Point (-15781065.09, -12244746.24) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119605842, end = 119607453, full = 15060790, total = 16622934, cells = 16 
Interval start = 119623613, end = 119689314, full = 15060790, total = 16622934, cells = 16 
Interval start = 119698684, end = 119738796, full = 15060790, total = 16622934, cells = 16 
   Point (-14463791.49, -1670512.64) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 119811561, end = 120617070, full = 15060790, total = 16622934, cells = 16 
   Point (11016068.95, 157250.03) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016089.07, 157278.84) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016069.01, 157337.89) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016089.07, 157369.76) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016069.01, 157420.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016980.28, 157485.45) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016068.93, 157519.46) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016980.28, 157605.27) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 120699724, end = 121585718, full = 15060790, total = 16622934, cells = 16 
   Point (481.65, 13444219.58) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (455.67, 2706262.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (439.80, 22655.48) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (441.52, 11579.36) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (424.62, 32258.75) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (414.32, 32429.89) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (407.12, 11685.62) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (419.09, 22497.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (421.03, 2695825.20) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 121659673, end = 122588799, full = 15060790, total = 16622934, cells = 16 
   Point (11016068.95, 157964.74) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016068.95, 157010.69) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016068.93, 157064.76) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016089.14, 157082.58) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016069.01, 157153.12) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016089.07, 157165.66) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016089.14, 157200.58) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016089.14, 157257.25) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11016980.28, 157321.51) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 122671044, end = 123647049, full = 15060790, total = 16622934, cells = 16 
   Point (243239.13, 470.50) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (243245.67, 488.38) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (243237.20, 484.80) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (243234.74, 483.78) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (243241.93, 492.92) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (243226.58, 493.71) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (243241.43, 490.14) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (243246.52, 485.97) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (243221.22, 480.05) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (243224.10, 478.03) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 123741675, end = 124603093, full = 15060790, total = 16622934, cells = 16 
   Point (5864819.04, -12460355.97) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (5727193.44, -12974158.21) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (6107302.24, -14101377.41) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (5580392.80, -14575858.05) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (5702945.12, -13981446.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (5134092.64, -14419227.01) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (6948129.12, -15864951.17) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (5211425.13, -19396030.85) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 124682798, end = 125518909, full = 15060790, total = 16622934, cells = 16 
   Point (386.09, 2717328.97) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (388.53, 22779.85) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (412.70, 11617.89) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (425.16, 22381.07) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (433.45, 2706840.56) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (440.23, 11344.54) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (447.99, 2706217.43) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (450.03, 22372.94) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (464.59, 13433529.37) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 125594171, end = 126463883, full = 15060790, total = 16622934, cells = 16 
   Point (4832627.04, -11932135.81) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (4831316.32, -10930090.37) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (3058567.52, -12240155.01) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (3835169.12, -11726352.77) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (3500935.52, -11433406.85) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (4495116.64, -11085410.69) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (4559997.28, -11906576.77) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (4137290.08, -12259815.81) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 126582491, end = 127436464, full = 15060790, total = 16622934, cells = 16 
   Point (2706243.31, -2867197.43) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (2718010.95, -2896033.27) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (2696038.14, 465963.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (2695997.84, 477104.65) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (2707101.99, 549849.61) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11673.97, 486932.49) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11617.03, 549191.69) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (11556.08, -1966079.99) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (13433407.52, 1048578.57) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 127553895, end = 128442590, full = 15060790, total = 16622934, cells = 16 
   Point (860.33, 3829880.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (168632.49, 3830590.45) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (168632.66, 3829942.80) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (168468.97, 21632.98) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (168468.99, 9866413.13) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (168632.49, 3829943.03) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (655.69, 21633.28) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (168468.98, 21633.08) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (655.53, 9866411.47) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Interval start = 128551794, end = 129319764, full = 15060790, total = 16622934, cells = 16 
   Point (1606945.12, -13190427.01) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (997460.32, -12993819.01) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (1522403.68, -13455192.45) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (2670594.40, -13392933.25) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (2305558.88, -12499677.57) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (2425489.76, -11671302.53) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
   Point (2627340.64, -11566444.93) not in rect [157980.00, 243480.00, 158270.00, 243770.00]
Read 16622934 points
Read 30595 points in bbox
adrienmichez commented 3 years ago

Hi,

I would like to use spatial indexing again but I'm afraid to experience this bug again and not see it.

Is there a way to test if the bug occurred in my process?

Jean-Romain commented 3 years ago

I'm closer and closer to the bug. I reached very deep lines of code but I'm not yet sure where does it come from. I cannot say anything about this bug expect that I'm sure the lax files is properly read. What happens is that after a certain number of file cursor moves it seems that there is a lag of few bits between where the cursor should be and where it actually is. It is windows specific and never reproduced with another dataset. Your dataset is very big and this does not help to figure out the issue.

It is not reproducible in all OS, not reproducible in other dataset, not reproducible with LAStools, not alway reproducible with another spatial query. My best guess so far is that it is specific to the c++ compiler used by R on windows but cannot explain why only with this dataset.

But I'm still investigating regularly getting closer and closer to the problem.

Jean-Romain commented 3 years ago

Good new I finally identified the issue :+1: . Now let try to fix it.

adrienmichez commented 3 years ago

Wow... Great news!

Jean-Romain commented 3 years ago

I fixed it :tada:.

That was a big issue actually and I'm surprised nobody reported it before. I'm tagging @bi0m3trics wondering if you already encountered such problem.

To make it simple the problem came from the fact that your file was more than ~2GB. This is not expected to be a problem to read such big file in the regular case. However when using a lax file, the reader can jump to different parts of the files. If the file is bigger than ~2GB it may try to jump to positions bigger than 2147483647. The problem was that the code badly handled the case when compiled with mingw which is the compiler used by R on windows. It assigned a method to read the file that cannot handle more than 2GB. The points that were missing in our query were those in the file after 2GB.

adrienmichez commented 3 years ago

Great Thanks for the investigation!

Jean-Romain commented 3 years ago

Thank you for reporting. Please try rlas 1.4.0 on your side and give me some feed back. I fixed the issue in fea7e62 but I will also made a full update of LASlib in rlas 1.4.0. So please stay informed. The current version 1.4.0 will be outdated in few hours/days

adrienmichez commented 3 years ago

I just tested... It's now working using lidR and rlas from github 🎉