search

connormanning / entwine

Entwine - point cloud organization for massive datasets
https://entwine.io
Other
451 stars 128 forks source link

Progress percentage incorrectly calculated #186

Closed GeoSpark closed 5 years ago

GeoSpark commented 5 years ago

When running the build command on a number of LAS files in several runs, subsequent runs after the first print the wrong progress percentage.

E.g. 50 LAS files built in runs of 20 produce this output on the second run:

# docker run -it --rm -v $(pwd):/upload -v /mnt/volume_lon1_01/entwine/:/entwine connormanning/entwine:latest build --threads 2 -i /upload/ -o /entwine/pc --run 20
Scanning input
Resolving [file]: /upload/* ...
        Resolved to 50 paths.
1/50: /upload/39998-MKF-RC-REVA-4978_1.las
2/50: /upload/39998-MKF-RC-REVA-4978_10.las
3/50: /upload/39998-MKF-RC-REVA-4978_11.las
4/50: /upload/39998-MKF-RC-REVA-4978_12.las
5/50: /upload/39998-MKF-RC-REVA-4978_13.las
6/50: /upload/39998-MKF-RC-REVA-4978_14.las
7/50: /upload/39998-MKF-RC-REVA-4978_15.las
8/50: /upload/39998-MKF-RC-REVA-4978_16.las
9/50: /upload/39998-MKF-RC-REVA-4978_17.las
10/50: /upload/39998-MKF-RC-REVA-4978_18.las
11/50: /upload/39998-MKF-RC-REVA-4978_19.las
12/50: /upload/39998-MKF-RC-REVA-4978_2.las
13/50: /upload/39998-MKF-RC-REVA-4978_20.las
14/50: /upload/39998-MKF-RC-REVA-4978_21.las
15/50: /upload/39998-MKF-RC-REVA-4978_22.las
16/50: /upload/39998-MKF-RC-REVA-4978_23.las
17/50: /upload/39998-MKF-RC-REVA-4978_24.las
18/50: /upload/39998-MKF-RC-REVA-4978_25.las
19/50: /upload/39998-MKF-RC-REVA-4978_26.las
20/50: /upload/39998-MKF-RC-REVA-4978_27.las
21/50: /upload/39998-MKF-RC-REVA-4978_28.las
22/50: /upload/39998-MKF-RC-REVA-4978_29.las
23/50: /upload/39998-MKF-RC-REVA-4978_3.las
24/50: /upload/39998-MKF-RC-REVA-4978_30.las
25/50: /upload/39998-MKF-RC-REVA-4978_31.las
26/50: /upload/39998-MKF-RC-REVA-4978_32.las
27/50: /upload/39998-MKF-RC-REVA-4978_33.las
28/50: /upload/39998-MKF-RC-REVA-4978_34.las
29/50: /upload/39998-MKF-RC-REVA-4978_35.las
30/50: /upload/39998-MKF-RC-REVA-4978_36.las
31/50: /upload/39998-MKF-RC-REVA-4978_37.las
32/50: /upload/39998-MKF-RC-REVA-4978_38.las
33/50: /upload/39998-MKF-RC-REVA-4978_39.las
34/50: /upload/39998-MKF-RC-REVA-4978_4.las
35/50: /upload/39998-MKF-RC-REVA-4978_40.las
36/50: /upload/39998-MKF-RC-REVA-4978_41.las
37/50: /upload/39998-MKF-RC-REVA-4978_42.las
38/50: /upload/39998-MKF-RC-REVA-4978_43.las
39/50: /upload/39998-MKF-RC-REVA-4978_44.las
40/50: /upload/39998-MKF-RC-REVA-4978_45.las
41/50: /upload/39998-MKF-RC-REVA-4978_46.las
42/50: /upload/39998-MKF-RC-REVA-4978_47.las
43/50: /upload/39998-MKF-RC-REVA-4978_48.las
44/50: /upload/39998-MKF-RC-REVA-4978_49.las
45/50: /upload/39998-MKF-RC-REVA-4978_5.las
46/50: /upload/39998-MKF-RC-REVA-4978_50.las
47/50: /upload/39998-MKF-RC-REVA-4978_6.las
48/50: /upload/39998-MKF-RC-REVA-4978_7.las
49/50: /upload/39998-MKF-RC-REVA-4978_8.las
50/50: /upload/39998-MKF-RC-REVA-4978_9.las
Determined SRS from an input file

Continuing previous index...

Entwine Version: 2.0.0
EPT Version: 1.0.0
Input:
        Files: 50
        Inserting up to 20 files
        Total points: 176,701,607
        Density estimate (per square unit): 2442.67
        Threads: [1, 4]
Output:
        Path: /entwine/pc/
        Data type: laszip
        Hierarchy type: json
        Sleep count: 2,097,152
        Scale: 0.001
        Offset: (3775273, -87274, 5122976)
Metadata:
        Bounds: [(3775257, -87328, 5122964), (3775289, -87219, 5122988)]
        Cube: [(3775217, -87330, 5122920), (3775329, -87218, 5123032)]
        Storing dimensions: [
                X:int32, Y:int32, Z:int32, Intensity:uint16, ReturnNumber:uint8,
                NumberOfReturns:uint8, ScanDirectionFlag:uint8,
                EdgeOfFlightLine:uint8, Classification:uint8, ScanAngleRank:float,
                UserData:uint8, PointSourceId:uint16, Red:uint16, Green:uint16,
                Blue:uint16, OriginId:uint32
        ]
Build parameters:
        Span: 256
        Resolution 2D: 256 * 256 = 65,536
        Resolution 3D: 256 * 256 * 256 = 16,777,216
        Maximum node size: 65,536
        Minimum node size: 16,384
        Cache size: 64

Adding 20 - /upload/39998-MKF-RC-REVA-4978_28.las
Adding 21 - /upload/39998-MKF-RC-REVA-4978_29.las
Adding 22 - /upload/39998-MKF-RC-REVA-4978_3.las
00:10 - 82% - 2,777,088 - 999(999)M/h - 0W - 56R - 91A
        Done 20
Adding 23 - /upload/39998-MKF-RC-REVA-4978_30.las
00:20 - 83% - 4,934,865 - 888(776)M/h - 56W - 44R - 114A
00:30 - 84% - 7,068,066 - 848(767)M/h - 85W - 34R - 94A
        Done 21
Adding 24 - /upload/39998-MKF-RC-REVA-4978_31.las
00:40 - 85% - 8,853,922 - 796(642)M/h - 26W - 59R - 156A
        Done 22
Adding 25 - /upload/39998-MKF-RC-REVA-4978_32.las
00:50 - 86% - 10,602,099 - 763(629)M/h - 133W - 29R - 83A
01:00 - 88% - 13,887,091 - 833(1,182)M/h - 13W - 39R - 165A
        Done 23
Adding 26 - /upload/39998-MKF-RC-REVA-4978_33.las
01:10 - 89% - 16,241,476 - 835(847)M/h - 107W - 34R - 129A
        Done 24
Adding 27 - /upload/39998-MKF-RC-REVA-4978_34.las
01:20 - 90% - 18,006,037 - 810(635)M/h - 100W - 33R - 83A
01:30 - 92% - 21,184,533 - 847(1,144)M/h - 0W - 39R - 180A
        Done 25
Adding 28 - /upload/39998-MKF-RC-REVA-4978_35.las
01:40 - 93% - 23,330,022 - 839(772)M/h - 119W - 29R - 127A
        Done 26
Adding 29 - /upload/39998-MKF-RC-REVA-4978_36.las
01:50 - 94% - 24,738,231 - 809(506)M/h - 107W - 30R - 76A
02:00 - 95% - 26,868,151 - 806(766)M/h - 4W - 49R - 160A
        Done 27
Adding 30 - /upload/39998-MKF-RC-REVA-4978_37.las
02:10 - 96% - 28,350,088 - 785(533)M/h - 126W - 27R - 83A
02:20 - 98% - 31,270,536 - 804(1,051)M/h - 0W - 39R - 169A
        Done 28
Adding 31 - /upload/39998-MKF-RC-REVA-4978_38.las
02:30 - 99% - 32,846,681 - 788(567)M/h - 125W - 33R - 104A
02:40 - 100% - 35,340,330 - 795(897)M/h - 14W - 46R - 188A
        Done 29
Adding 32 - /upload/39998-MKF-RC-REVA-4978_39.las
02:50 - 101% - 36,454,442 - 771(401)M/h - 119W - 47R - 125A
03:00 - 102% - 38,874,363 - 777(871)M/h - 23W - 47R - 191A
        Done 30
Adding 33 - /upload/39998-MKF-RC-REVA-4978_4.las
03:10 - 103% - 40,209,659 - 761(480)M/h - 121W - 35R - 118A
03:20 - 104% - 42,408,396 - 763(791)M/h - 96W - 29R - 91A
        Done 31
Adding 34 - /upload/39998-MKF-RC-REVA-4978_40.las
03:30 - 105% - 45,054,412 - 772(952)M/h - 21W - 38R - 149A
        Done 32
Adding 35 - /upload/39998-MKF-RC-REVA-4978_41.las
03:40 - 107% - 46,950,045 - 768(682)M/h - 117W - 33R - 95A
03:50 - 108% - 49,476,462 - 774(909)M/h - 33W - 37R - 138A
        Done 33
Adding 36 - /upload/39998-MKF-RC-REVA-4978_42.las
04:00 - 109% - 51,635,054 - 774(777)M/h - 70W - 35R - 135A
        Done 34
Adding 37 - /upload/39998-MKF-RC-REVA-4978_43.las
04:10 - 110% - 53,022,783 - 763(499)M/h - 120W - 35R - 76A
04:20 - 111% - 55,140,415 - 763(762)M/h - 0W - 53R - 166A
        Done 35
Adding 38 - /upload/39998-MKF-RC-REVA-4978_44.las
04:30 - 112% - 56,544,528 - 753(505)M/h - 147W - 31R - 71A
04:40 - 113% - 58,760,464 - 755(797)M/h - 0W - 55R - 165A
        Done 36
Adding 39 - /upload/39998-MKF-RC-REVA-4978_45.las
04:50 - 114% - 60,291,553 - 748(551)M/h - 150W - 25R - 73A
05:00 - 115% - 62,343,649 - 748(738)M/h - 0W - 59R - 166A
        Done 37
        Pushes complete - joining...
05:10 - 116% - 63,612,594 - 738(456)M/h - 142W - 29R - 79A
05:20 - 117% - 65,554,098 - 737(698)M/h - 9W - 60R - 157A
05:30 - 118% - 67,146,627 - 732(573)M/h - 136W - 30R - 89A
        Done 38
05:40 - 119% - 69,415,811 - 734(816)M/h - 20W - 56R - 152A
        Done 39
Reawakened: 1354
Saving registry...
05:50 - 120% - 70,680,660 - 727(455)M/h - 174W - 22R - 28A
Saving metadata...

Index completed in 05:51.
Save complete.
        Points inserted:
                Previously: 70,680,660
                Currently:  70,680,660
                Total:      141,361,320