search

tmontaigu / pylas

⚠️ pylas was merged into laspy 2.0 https://github.com/laspy/laspy⚠️
BSD 3-Clause "New" or "Revised" License
39 stars 13 forks source link

Buffer smaller than requested size for some LAS files #12

Closed pmaggiore closed 4 years ago

pmaggiore commented 4 years ago

Hi, I'm trying to read a received LAS file with pylas and I'm getting this:

>>> import pylas
>>> f1 = pylas.open('/usr/src/data/pylas/lidar/pointcloud.las')
>>> f1.read()
Traceback (most recent call last):
    File "/opt/anaconda3/lib/python3.7/site-packages/pylas-0.3.4-py3.7.egg/pylas/point/record.py", line 260, in from_stream
        data = np.frombuffer(point_data_buffer, dtype=points_dtype, count=count)
ValueError: buffer is smaller than requested size

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/opt/anaconda3/lib/python3.7/site-packages/pylas-0.3.4-py3.7.egg/pylas/lasreader.py", line 62, in read
        points = self._read_points(vlrs)
    File "/opt/anaconda3/lib/python3.7/site-packages/pylas-0.3.4-py3.7.egg/pylas/lasreader.py", line 115, in _read_points
        self.stream, point_format, self.header.point_count
    File "/opt/anaconda3/lib/python3.7/site-packages/pylas-0.3.4-py3.7.egg/pylas/point/record.py", line 269, in from_stream
        points_dtype,
    File "/opt/anaconda3/lib/python3.7/site-packages/pylas-0.3.4-py3.7.egg/pylas/point/record.py", line 32, in raise_not_enough_bytes_error
        point_data_buffer_len / points_dtype.itemsize,
pylas.errors.PylasError: The file does not contain enough bytes to store the expected number of points
expected 230066000 bytes, read 195556100 bytes (34509900 bytes missing == 862747.5 points) and it cannot be corrected
195556100 (bytes) / 40 (point_size) = 4888902.5 (points)

I can't attach the LAS file that is bigger than 10MB but I could send a link to download it if needed. I hope you can help.

pm

tmontaigu commented 4 years ago

This error is due to the header indicating a number of points that does not match the reality.

Do other softwares (lasinfo, cloudcompare, etc) manage to open the file without error or without printing a warning ?

If you can seend a link with the file that would help the debug

pmaggiore commented 4 years ago

Hi @tmontaigu, thank you for your help.

I'm also sending the output of lasinfo: `--------------------------------------------------------- Header Summary

Version: 1.2 Source ID: 0 Reserved: 0 Project ID/GUID: '00000000-0000-0000-0000-000000000000' System ID: 'EXTRACTION' Generating Software: 'LASfunctions v.1.0' File Creation Day/Year: 156/2017 Header Byte Size 227 Data Offset: 2145 Header Padding: 0 Number Var. Length Records: 5 Point Data Format: 3 Number of Point Records: 2023834 Compressed: False Number of Points by Return: 1505024 444251 69170 5187 201 Scale Factor X Y Z: 0.00025000000000 0.00025000000000 0.00025000000000 Offset X Y Z: 465864.0000 5060643.0000 355.0000 Min X Y Z: 465851.8133 5060638.0048 334.1078 Max X Y Z: 466398.7130 5060851.2268 374.3673 Spatial Reference:
Reference defined, but GDAL is not available for WKT support

Geotiff_Information: Version: 1 Key_Revision: 1.0 Tagged_Information: End_Of_Tags. Keyed_Information: GTModelTypeGeoKey (Short,1): ModelTypeProjected GTRasterTypeGeoKey (Short,1): RasterPixelIsPoint GTCitationGeoKey (Ascii,28): "WGS 84 / UTM zone 32N_ortho" GeographicTypeGeoKey (Short,1): User-Defined GeogCitationGeoKey (Ascii,6): "WGS84" GeogGeodeticDatumGeoKey (Short,1): User-Defined GeogPrimeMeridianGeoKey (Short,1): User-Defined GeogLinearUnitsGeoKey (Short,1): Linear_Meter GeogAngularUnitsGeoKey (Short,1): Angular_Degree GeogEllipsoidGeoKey (Short,1): User-Defined GeogSemiMajorAxisGeoKey (Double,1): 6378137
GeogInvFlatteningGeoKey (Double,1): 298.257223563
GeogAzimuthUnitsGeoKey (Short,1): Angular_Degree GeogPrimeMeridianLongGeoKey (Double,1): 0
ProjectedCSTypeGeoKey (Short,1): User-Defined PCSCitationGeoKey (Ascii,28): "WGS 84 / UTM zone 32N_ortho" ProjectionGeoKey (Short,1): User-Defined ProjCoordTransGeoKey (Short,1): CT_TransverseMercator ProjLinearUnitsGeoKey (Short,1): Linear_Meter ProjNatOriginLongGeoKey (Double,1): 9
ProjNatOriginLatGeoKey (Double,1): 0
ProjFalseEastingGeoKey (Double,1): 500000
ProjFalseNorthingGeoKey (Double,1): 0
ProjScaleAtNatOriginGeoKey (Double,1): 0.9996
VerticalUnitsGeoKey (Short,1): Linear_Meter End_Of_Keys. End_Of_Geotiff.

VLR Summary

User: 'LASF_Projection' - Description: 'GeoKeyDirectoryTag (mandatory)'
ID: 34735 Length: 208 Total Size: 262
User: 'LASF_Projection' - Description: 'GeoDoubleParamsTag (optional)'
ID: 34736 Length: 64 Total Size: 118
User: 'LASF_Projection' - Description: 'GeoASCIIParamsTag (optional)'
ID: 34737 Length: 62 Total Size: 116
User: 'LASF_Projection' - Description: 'OGC COORDINATE SYSTEM WKT'
ID: 2112 Length: 738 Total Size: 792
User: 'LASF_Spec' - Description: 'RIEGL Extra Bytes'
ID: 4 Length: 576 Total Size: 630

Schema Summary

Point Format ID: 3 Number of dimensions: 16 Custom schema?: false Size in bytes: 34

Dimensions

'X' -- size: 32 offset: 0 'Y' -- size: 32 offset: 4 'Z' -- size: 32 offset: 8 'Intensity' -- size: 16 offset: 12 'Return Number' -- size: 3 offset: 14 'Number of Returns' -- size: 3 offset: 14 'Scan Direction' -- size: 1 offset: 14 'Flightline Edge' -- size: 1 offset: 14 'Classification' -- size: 8 offset: 15 'Scan Angle Rank' -- size: 8 offset: 16 'User Data' -- size: 8 offset: 17 'Point Source ID' -- size: 16 offset: 18 'Time' -- size: 64 offset: 20 'Red' -- size: 16 offset: 28 'Green' -- size: 16 offset: 30 'Blue' -- size: 16 offset: 32

Point Inspection Summary

Header Point Count: 2023834 Actual Point Count: 2023834

Minimum and Maximum Attributes (min,max)

Min X, Y, Z: 465851.8133, 5060638.0048, 334.1078 Max X, Y, Z: 466398.7130, 5060851.2268, 374.3673 Bounding Box: 465851.8133, 5060638.0048, 466398.7130, 5060851.2268 Time: 313666.295394, 313691.616049 Return Number: 1, 6 Return Count: 1, 6 Flightline Edge: 0, 0 Intensity: 0, 54197 Scan Direction Flag: 1, 1 Scan Angle Rank: -43, 9 Classification: 2, 3 Point Source Id: 0, 0 User Data: 0, 0 Minimum Color (RGB): 0 1024 0 Maximum Color (RGB): 65280 65280 65280

Number of Points by Return

(1) 1505024 (2) 444251  (3) 69170   (4) 5187    (5) 201 (6) 1

Number of Returns by Pulse

(1) 1035210 (2) 762013  (3) 203503  (4) 21983   (5) 1115    (6) 10

Point Classifications

243759 Ground (2) 
1780075 Low Vegetation (3) 
0 withheld
0 keypoint
0 synthetic

`

I'd like to add that I'm able to read this las file with libLAS library without problems.

I hope these infos help identifying the problem. Thank you for helping

pm

tmontaigu commented 4 years ago

The problem comes from the fact that in the LAS header the point size is said to be 34 (the point size for a LAS with point format 3 and no extra bytes) but there is also an Extra bytes vlr present in the file:

lasinfo reports

variable length header record 5 of 5:
  reserved             43707
  user ID              'LASF_Spec'
  record ID            4
  length after header  576
  description          'RIEGL Extra Bytes'
    Extra Byte Descriptions
      data type: 3 (unsigned short), name "Amplitude", description: "Echo signal amplitude [dB]", min: 0, max: 10000, scale: 0.01, offset: 0 (not set)

      data type: 4 (short), name "Reflectance", description: "Echo signal reflectance [dB]", min: -5000, max: 15000, scale: 0.01, offset: 0 (not set)

      data type: 3 (unsigned short), name "Deviation", description: "Pulse shape deviation", min: 0, max: 32767, scale: 1 (not set), offset: 0 (not set)

which brings the point_size to 40 bytes. The files has 2_023_834 points and is 68_812_501 bytes on disk.

2_023_834 40 = ‭‭80,953,360‬ bytes 2_023_834 34 = ‭68,810,356‬ bytes

Do you expect the fields 'Amplitude', 'Reflectance', 'Deviation' to be in that particular file ?

pmaggiore commented 4 years ago

Thank you very much for the explanation! The fact is that I just need to read the points from the file; I don't need those fields (amplitude, reflectance, deviation) to be considered for the data extraction I'm going to do.

Is there a way to be able to get all the points anyway?

Thank you again,

pm

tmontaigu commented 4 years ago

Well you can either reinstall pylas from github since i pushed a commit that fixes this, or wait for a new release. or you could use lastools' las2las to remove the wrong vlr

las2las -remove_vlr 4 -i pointcloud.las -o pointcloud_without_extra_bytes_vlr.las
pmaggiore commented 4 years ago

Wow, perfect! Thanks for the quick responses and fix :)