search

makingglitches / CensusScripts

Contains miners and scripts for working with census file data and boundaries. Getting started at least, actively would have been finished and developed if these people didnt deny the passage of time and keep stealing my computer and code.
1 stars 0 forks source link

Compression size Test #4

Closed makingglitches closed 3 years ago

makingglitches commented 3 years ago

I need a GDAL compression comparison test.

ERDAS Imagine rasters are incredibly storage intensive.

A "small" sample I have encoded to 30m per pixel, is 20 GB in size. It contains one pallette layer. That's it.

I need a test which will measure just compression size difference at the following tile sizes:

512x512 (native) 1024x1024

I need the test to use compressed TIFF and PNG and to cover the entire raster.

Space differences shall be calculated by iterating through the image data, using the highest compression level possible, and writing a seperate file per tile sampled, recording its size, deleting it, and moving on to the next.

Also of interest is compression time on the host system, though this would need to be benchmarked across several systems to be of great use.

makingglitches commented 3 years ago

would be great if they had let me finish this instead of deleting everything and starting over, over and over again because they're loser chomos that sold their whole lives downriver and can't stand anyone progressing in their life at a normal rate in any sense and fucked up the whole economy and corrupted the whole government to be non functioning being the fast replicating pile of trash that they are.

makingglitches commented 3 years ago

Its like giving a bazooka to one of the kids in the village of the damned all grown up.

makingglitches commented 3 years ago

Findings as follows.

Processing Tile: 314 x 203 #64260Writing Variance Data To File.
Calculation of Variances took: 00:23:19.8368665
At earlier test speed, encoding to two formats took a second each.
For the present raster that equals 17.85 hours.
To reduce this down to an hour. Record count must reduce to: 3600 of 64260
Dividing stdev range into panels, panel size was: 1.27500278487556
Compression Size Test For Specific Raster
Present Filename: C:\Users\John\Documents\CensusProject\CensusShapeFileData\TreeCanopy\nlcd_2016_treecanopy_2019_08_31\nlcd_2016_treecanopy_2019_08_31.img
Size: 20620 Mb
Running tests for 3644 selections
Encoding for tile size: 512

Panel Start Panel End Copy Time Png Time Tiff Time PNG Size Tiff Size
0 1.27500278487556 07:35:18 00:04:31 00:05:48 75254189 110386327
1.27500278487556 2.55000556975112 00:57:38 00:00:19 00:00:31 18016040 26359701
2.55000556975112 3.82500835462668 00:25:13 00:00:15 00:00:21 21110052 30802459
3.82500835462668 5.10001113950223 00:13:10 00:00:07 00:00:10 22109063 33385283
5.10001113950223 6.37501392437779 00:10:37 00:00:05 00:00:08 23172655 36920368
6.37501392437779 7.65001670925335 00:08:01 00:00:07 00:00:08 26926189 44309126
7.65001670925335 8.92501949412891 00:09:22 00:00:09 00:00:13 42711408 73113992
8.92501949412891 10.2000222790045 00:09:15 00:00:08 00:00:11 40442923 71068275
10.2000222790045 11.47502506388 00:09:23 00:00:08 00:00:09 46640604 83346291
11.47502506388 12.7500278487556 00:08:12 00:00:11 00:00:14 77215494 143604392
12.7500278487556 14.0250306336311 00:07:59 00:00:12 00:00:15 79848672 151787260
14.0250306336311 15.3000334185067 00:09:57 00:00:11 00:00:15 110377010 208678995
15.3000334185067 16.5750362033823 00:11:02 00:00:14 00:00:18 126952323 245715246
16.5750362033823 17.8500389882578 00:09:54 00:00:11 00:00:13 90459908 175526404
17.8500389882578 19.1250417731334 00:09:31 00:00:17 00:00:19 140695931 284630852
19.1250417731334 20.4000445580089 00:09:25 00:00:16 00:00:20 137012762 270742159
20.4000445580089 21.6750473428845 00:08:42 00:00:15 00:00:20 137008817 274502619
21.6750473428845 22.9500501277601 00:09:36 00:00:15 00:00:19 149012960 302293498
22.9500501277601 24.2250529126356 00:08:39 00:00:16 00:00:21 160054988 320484255
24.2250529126356 25.5000556975112 00:09:23 00:00:18 00:00:23 174123304 352360999
25.5000556975112 26.7750584823867 00:12:43 00:00:16 00:00:21 161089948 332175398
26.7750584823867 28.0500612672623 00:11:05 00:00:15 00:00:21 166995960 341705620
28.0500612672623 29.3250640521378 00:10:13 00:00:17 00:00:23 200873904 416290274
29.3250640521378 30.6000668370134 00:08:16 00:00:17 00:00:21 166163026 345664893
30.6000668370134 31.875069621889 00:09:49 00:00:18 00:00:23 194440246 407600287
31.875069621889 33.1500724067645 00:08:42 00:00:18 00:00:24 197530509 416633172
33.1500724067645 34.4250751916401 00:11:14 00:00:23 00:00:32 207039990 438181800
34.4250751916401 35.7000779765156 00:11:41 00:00:25 00:00:32 254577074 548214590
35.7000779765156 36.9750807613912 00:13:53 00:00:36 00:00:36 272844479 582755768
36.9750807613912 38.2500835462668 00:12:01 00:00:25 00:00:31 220918847 476599855
38.2500835462668 39.5250863311423 00:12:10 00:00:23 00:00:28 212640052 463812365
39.5250863311423 40.8000891160179 00:08:13 00:00:19 00:00:24 167309585 367726386
40.8000891160179 42.0750919008934 00:05:04 00:00:12 00:00:14 103280742 224707545
42.0750919008934 43.350094685769 00:02:15 00:00:04 00:00:06 40449813 85255527
43.350094685769 44.6250974706446 00:01:13 00:00:01 00:00:02 13803672 27514632
44.6250974706446 45.9001002555201 00:00:32 00:00:00 00:00:00 6648553 12377475
45.9001002555201 47.1751030403957 00:00:31 00:00:00 00:00:00 6027052 11733376
47.1751030403957 48.4501058252712 00:00:15 00:00:00 00:00:00 4059982 7837256 48.4501058252712 49.7251086101468 00:00:17 00:00:00 00:00:00 2022119 4096432 49.7251086101468 51.0001113950224 00:00:16 00:00:00 00:00:00 180124 311661
51.0001113950224 52.2751141798979 00:00:17 00:00:00 00:00:00 38877 57012
52.2751141798979 53.5501169647735 00:00:17 00:00:00 00:00:00 2063584 4015464 53.5501169647735 54.825119749649 00:00:16 00:00:00 00:00:00 2615042 5374920 54.825119749649 56.1001225345246 00:00:17 00:00:00 00:00:00 81681 129019
56.1001225345246 57.3751253194002 00:00:17 00:00:00 00:00:00 168575 300379
57.3751253194002 58.6501281042757 00:00:16 00:00:00 00:00:00 458477 715642
58.6501281042757 59.9251308891513 00:00:17 00:00:00 00:00:00 38877 57012
59.9251308891513 61.2001336740268 00:00:17 00:00:00 00:00:00 38877 57012
61.2001336740268 62.4751364589024 00:00:16 00:00:00 00:00:00 2875063 5311588 62.4751364589024 63.750139243778 00:00:16 00:00:00 00:00:00 38877 57012
63.750139243778 65.0251420286535 00:00:15 00:00:00 00:00:00 38877 57012
65.0251420286535 66.3001448135291 00:00:11 00:00:00 00:00:00 1417914 2821156 66.3001448135291 67.5751475984046 00:00:11 00:00:00 00:00:00 38877 57012
67.5751475984046 68.8501503832802 00:00:12 00:00:00 00:00:00 61832 101031
68.8501503832802 70.1251531681557 00:00:11 00:00:00 00:00:00 38877 57012
70.1251531681557 71.4001559530313 00:00:12 00:00:00 00:00:00 38877 57012
71.4001559530313 72.6751587379068 00:00:11 00:00:00 00:00:00 66723 98210
72.6751587379068 73.9501615227824 00:00:13 00:00:00 00:00:00 3516914 6774253 73.9501615227824 75.2251643076579 00:00:10 00:00:00 00:00:00 60422 100352
75.2251643076579 76.5001670925335 00:00:12 00:00:00 00:00:00 1302764 2156601 76.5001670925335 77.7751698774091 00:00:10 00:00:00 00:00:00 38877 57012
77.7751698774091 79.0501726622846 00:00:11 00:00:00 00:00:00 68990 121808
79.0501726622846 80.3251754471602 00:00:12 00:00:00 00:00:00 38877 57012
80.3251754471602 81.6001782320357 00:00:12 00:00:00 00:00:00 38877 57012
81.6001782320357 82.8751810169113 00:00:14 00:00:00 00:00:00 1422002 2402788 82.8751810169113 84.1501838017868 00:00:24 00:00:00 00:00:00 2109048 3783057 84.1501838017868 85.4251865866624 00:00:11 00:00:00 00:00:00 38877 57012
85.4251865866624 86.7001893715379 00:00:11 00:00:00 00:00:00 3324044 5859476 86.7001893715379 87.9751921564135 00:00:12 00:00:00 00:00:00 38877 57012
87.9751921564135 89.250194941289 00:00:24 00:00:00 00:00:00 1364846 1870251 89.250194941289 90.5251977261646 00:00:11 00:00:00 00:00:00 59333 89856
90.5251977261646 91.8002005110401 00:00:12 00:00:00 00:00:00 582748 1127941 91.8002005110401 93.0752032959157 00:00:12 00:00:00 00:00:00 38877 57012
93.0752032959157 94.3502060807913 00:00:10 00:00:00 00:00:00 38877 57012
94.3502060807913 95.6252088656668 00:00:12 00:00:00 00:00:00 40269 62546
95.6252088656668 96.9002116505424 00:00:11 00:00:00 00:00:00 1270348 2572934 96.9002116505424 98.1752144354179 00:00:12 00:00:00 00:00:00 56102 109956
98.1752144354179 99.4502172202935 00:00:11 00:00:00 00:00:00 38877 57012
99.4502172202935 100.725220005169 00:00:12 00:00:00 00:00:00 38877 57012
100.725220005169 102.000222790045 00:00:11 00:00:00 00:00:00 38877 57012
102.000222790045 103.27522557492 00:00:24 00:00:00 00:00:00 2075776 3968626 103.27522557492 104.550228359796 00:00:24 00:00:00 00:00:00 1915786 3881910 104.550228359796 105.825231144671 00:00:12 00:00:00 00:00:00 300433 490625
105.825231144671 107.100233929547 00:00:12 00:00:00 00:00:00 3117109 6557982 107.100233929547 108.375236714422 00:00:11 00:00:00 00:00:00 2587732 5262786 108.375236714422 109.650239499298 00:00:11 00:00:00 00:00:00 56513 93748
109.650239499298 110.925242284173 00:00:26 00:00:00 00:00:00 77754 114025
110.925242284173 112.200245069049 00:00:23 00:00:00 00:00:00 3737147 7526773 112.200245069049 113.475247853925 00:00:22 00:00:00 00:00:00 196867 271677
113.475247853925 114.7502506388 00:00:25 00:00:00 00:00:00 77754 114025
114.7502506388 116.025253423676 00:00:23 00:00:00 00:00:00 2466013 4577525 116.025253423676 117.300256208551 00:00:26 00:00:00 00:00:00 96479 152581
117.300256208551 118.575258993427 00:00:24 00:00:00 00:00:00 2346953 4741959 118.575258993427 119.850261778302 00:00:23 00:00:00 00:00:00 797270 1205624 119.850261778302 121.125264563178 00:00:24 00:00:00 00:00:00 80967 139587
121.125264563178 122.400267348053 00:00:23 00:00:00 00:00:00 1063199 1987918 122.400267348053 123.675270132929 00:00:41 00:00:00 00:00:00 192083 305199
123.675270132929 124.950272917805 00:00:38 00:00:00 00:00:00 274176 435575
124.950272917805 126.22527570268 00:00:59 00:00:00 00:00:00 1401421 2557797 126.22527570268 127.500278487556 00:01:00 00:00:00 00:00:00 515865 857192
127.500278487556 128.775281272431 00:00:12 00:00:00 00:00:00 38877 57012
Process finished. 14:18:42 00:12:54 00:16:34 4347143650 8843438403
4145.75924873352 8433.76007366181 <==MB 16474.2407512665 12186.2399263382 <==Remaining 0.798944750303903 0.59099126703871 <==Percentage Saved

Concerning part is how long gdal is taking to load relatively small tiles into the bitmap. By randomly selecting from a sampling frame of panels the average compressions sizes were calculated in a stratified sample that measured the differences in variance in each panel.

Converting to what the entire process would take (record wise) by the inverse operation used to select the panels, tiff compression using .net 4.8 does not take much longer across 64000 some records, however it does not come anywhere near outperforming png.

I do not know if scale tiling is being used in .net, which is supposedly a feature of tiff images. This could greatly affect a few of the tested performances when accessing raster data.

However that being said gdal in .net is taking up the bulk of the operation above at the native tile size for the raster in question, which is a relatively simple single layer one.

Compression saves 79% for png speaking leaps and bounds to using this as the format.

If the scaling, load, and read operations perform well there is reason to recover space. This isnt NASA or Military Intelligence my scripts are looking to power !