search

pgpointcloud / pointcloud

A PostgreSQL extension for storing point cloud (LIDAR) data.
https://pgpointcloud.github.io/pointcloud/
Other
399 stars 108 forks source link

Allow RLE and SIGBITS compression for double values #60

Open strk opened 9 years ago

strk commented 9 years ago

Is there a good reason for double values not being compressible by anything but zlib ?

pramsey commented 9 years ago

The way IEEE double represents things you don't tend to get either repeated values or lots of shared bits, so they never pass the "is this any good?" test for the RLE and sigbits methods.

strk commented 9 years ago

The test is not performed at all for double values: https://github.com/pgpointcloud/pointcloud/blob/master/lib/pc_dimstats.c#L124

GEOS has a common-bits remover class that works with IEEE http://geos.osgeo.org/doxygen/classgeos_1_1precision_1_1CommonBitsRemover.html

pramsey commented 9 years ago

You know more than me then! :)

 

strk commented 9 years ago

I just wanted to know if there was a reason for that conditional, basically.

Remi-C commented 9 years ago

In theory it is possible to remove common bit from double by factorizing exponent part. Now taking in common the mantissa shouldn't work.

I would be very surpised to get a compression better than zlib with that.

Cheers, Rémi-C

2015-04-01 20:58 GMT+02:00 Sandro Santilli notifications@github.com:

I just wanted to know if there was a reason for that conditional, basically.

— Reply to this email directly or view it on GitHub https://github.com/pgpointcloud/pointcloud/issues/60#issuecomment-88594918 .

strk commented 9 years ago

RLE seems to work fine (so just needs to be unleashed), sigbits doesn't really work as it's written.

strk commented 9 years ago

uhm, there's actually something wrong with RLE in general, not directly related to the dimension being a double. Will file another ticket for that.

strk commented 9 years ago

I found that FLOAT values are currently candidated for use with both sigbits and rle. It's only DOUBLE that are rejected.

strk commented 9 years ago

The test added in 667dcf3ec0f734e6f498864164add8a3c37b3e20 shows that both RLE and SIGBITS encoding are capable of losslessly transform floating point values.

Remi-C commented 9 years ago

Out of pure curiosity, does it compress effectively float? Cheers

2015-04-13 17:11 GMT+02:00 Sandro Santilli notifications@github.com:

The test added in 667dcf3 https://github.com/pgpointcloud/pointcloud/commit/667dcf3ec0f734e6f498864164add8a3c37b3e20 shows that both RLE and SIGBITS encoding are capable of losslessly transform floating point values.

— Reply to this email directly or view it on GitHub https://github.com/pgpointcloud/pointcloud/issues/60#issuecomment-92394663 .

strk commented 9 years ago

You tell me (I'm off for today). PC_Compress() away your patches with floating points !