search

jeff-regier / Celeste.jl

Scalable inference for a generative model of astronomical images
MIT License
183 stars 28 forks source link

WIP : Joint auto batchsize 2 #646

Closed agnusmaximus closed 7 years ago

agnusmaximus commented 7 years ago

Benchmarks

9.0, 9.125, 24.0, 24.125

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [82.4469, 84.2131, 82.5839, 82.186, 82.8814, 82.8109, 81.9356, 82.2468]
[1]<1> INFO: Batch 1 avg threads idle: 2% (1.5499915712500005 / 84.213067005)
[1]<1> INFO: Batch 2 - [1.73308, 1.80646, 1.6067e-5, 2.1141e-5, 1.3663e-5, 1.7383e-5, 1.7516e-5, 1.79e-5]
[1]<1> INFO: Batch 2 avg threads idle: 76% (1.364000322625 / 1.806455537)
[1]<1> INFO: Batch 1 - [4.71484, 4.54403, 4.50113, 4.54192, 4.63044, 4.56471, 4.51283, 4.52111]
[1]<1> INFO: Batch 1 avg threads idle: 3% (0.14846715212499983 / 4.714842944)
[1]<1> INFO: Batch 2 - [1.6664e-5, 1.2367e-5, 0.129022, 1.8934e-5, 2.1737e-5, 1.4594e-5, 0.119348, 1.0276e-5]
[1]<1> INFO: Batch 2 avg threads idle: 76% (0.09796365737500001 / 0.129021654)
[1]<1> INFO: Batch 1 - [3.58585, 3.7005, 3.63955, 3.58123, 3.54887, 3.55051, 3.63424, 3.5795]
[1]<1> INFO: Batch 1 avg threads idle: 3% (0.0979735375000001 / 3.70050458)
[1]<1> INFO: Batch 2 - [1.6882e-5, 0.115387, 1.4115e-5, 1.8683e-5, 0.12338, 8.332e-6, 9.727e-6, 1.1774e-5]
[1]<1> INFO: Batch 2 avg threads idle: 76% (0.09352396925 / 0.123379746)
[1]<1> INFO: Total idle time: 26.815361681000006
[1]<1> 13:40:51.953: (active,inactive) pixels processed: (14212390,155308)
ERROR: LoadError: FileIO.File{FileIO.DataFormat{:UNKNOWN}}("/global/u1/a/agnusmax/Celeste.jl/celeste-9.0000-9.1250-24.0000-24.1250-rank1.jld") couldn't be recognized by FileIO.

Constant

[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [84.8036, 81.3766, 83.1119, 81.2999, 84.3911, 80.9158, 94.2476, 81.8078]
[1]<1> INFO: Batch 1 avg threads idle: 11%
[1]<1> INFO: Batch 1 - [3.83728, 3.81591, 3.91635, 4.93319, 3.71564, 3.77225, 3.8513, 3.83542]
[1]<1> INFO: Batch 1 avg threads idle: 20%
[1]<1> INFO: Batch 1 - [3.55842, 3.48159, 3.428, 4.63811, 3.47543, 3.50152, 3.56348, 3.47533]
[1]<1> INFO: Batch 1 avg threads idle: 22%
[1]<1> 14:16:27.194: (active,inactive) pixels processed: (14060987,152692)
ERROR: LoadError: FileIO.File{FileIO.DataFormat{:UNKNOWN}}("/global/u1/a/agnusmax/Celeste.jl/celeste-9.0000-9.1250-24.0000-24.1250-rank1.jld") couldn't be recognized by FileIO.

309.5625, 309.625, 3.375, 3.4375

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [73.6423, 73.2383, 72.9768, 73.8945, 72.423, 72.226, 72.9467, 75.8665]
[1]<1> INFO: Batch 1 avg threads idle: 3% (2.464758524249998 / 75.866523723)
[1]<1> INFO: Batch 2 - [5.1524, 5.78035, 5.72584, 5.70007, 4.84464, 5.989, 5.28996, 4.90049]
[1]<1> INFO: Batch 2 avg threads idle: 9% (0.5661555035000003 / 5.988998611)
[1]<1> INFO: Batch 1 - [3.82054, 3.87856, 5.1667, 3.96353, 3.78314, 3.92551, 3.95573, 4.93313]
[1]<1> INFO: Batch 1 avg threads idle: 19% (0.9883429235000001 / 5.166698032)
[1]<1> INFO: Batch 2 - [0.552809, 0.524507, 0.567617, 0.517187, 0.445432, 0.487252, 0.477587, 0.527442]
[1]<1> INFO: Batch 2 avg threads idle: 10% (0.05513793424999999 / 0.567617084)
[1]<1> INFO: Batch 1 - [2.5776, 2.50414, 2.46454, 2.38224, 2.40925, 2.44333, 2.4369, 2.39545]
[1]<1> INFO: Batch 1 avg threads idle: 5% (0.12592218324999987 / 2.577604132)
[1]<1> INFO: Batch 2 - [0.360041, 0.392739, 0.440091, 0.429864, 0.372701, 0.389092, 0.417945, 0.373818]
[1]<1> INFO: Batch 2 avg threads idle: 10% (0.043054297125000016 / 0.440090782)
[1]<1> INFO: Total idle time: 33.94697092699999
[1]<1> 13:43:21.284: (active,inactive) pixels processed: (12417533,1052052)
123.104016 seconds (59.53 M allocations: 3.300 GiB, 2.28% gc time)

Constant

[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [68.7134, 68.5477, 68.3564, 69.8915, 68.8127, 68.6654, 68.3029, 68.5504]
[1]<1> INFO: Batch 1 avg threads idle: 2%
[1]<1> INFO: Batch 1 - [2.48133, 2.46365, 2.58005, 2.47839, 2.52042, 2.4499, 2.42532, 2.49737]
[1]<1> INFO: Batch 1 avg threads idle: 4%
[1]<1> INFO: Batch 1 - [2.33157, 2.34596, 2.38669, 2.46326, 2.34625, 2.363, 2.36467, 2.59745]
[1]<1> INFO: Batch 1 avg threads idle: 8%
[1]<1> 14:20:53.535: (active,inactive) pixels processed: (8922180,95504)
101.102288 seconds (47.13 M allocations: 3.502 GiB, 1.66% gc time)

29.0, 29.125, 13.75, 13.875

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [61.7948, 62.1873, 60.7879, 62.4327, 60.3007, 60.9557, 62.2322, 60.5253]
[1]<1> INFO: Batch 1 avg threads idle: 2% (1.0306428075000005 / 62.432733356)
[1]<1> INFO: Batch 2 - [12.3111, 12.1952, 11.7234, 12.6688, 11.8635, 13.8281, 11.5953, 12.1345]
[1]<1> INFO: Batch 2 avg threads idle: 11% (1.5380799413750001 / 13.828067699)
[1]<1> INFO: Batch 1 - [1.54144, 1.59813, 1.61608, 1.55517, 1.57427, 1.60552, 1.64728, 1.67119]
[1]<1> INFO: Batch 1 avg threads idle: 4% (0.07005407737500005 / 1.671187503)
[1]<1> INFO: Batch 2 - [0.96167, 0.970166, 0.982189, 0.938401, 1.13168, 1.07885, 0.970967, 0.910064]
[1]<1> INFO: Batch 2 avg threads idle: 12% (0.13867934400000004 / 1.13167689)
[1]<1> INFO: Batch 1 - [1.43709, 1.41183, 1.44953, 1.40237, 1.3969, 1.45887, 1.53149, 1.52416]
[1]<1> INFO: Batch 1 avg threads idle: 5% (0.07996445312500006 / 1.531493213)
[1]<1> INFO: Batch 2 - [0.884311, 0.945527, 0.921114, 0.949863, 0.924633, 1.10944, 0.925392, 0.946918]
[1]<1> INFO: Batch 2 avg threads idle: 14% (0.1585398252500001 / 1.109439486)
[1]<1> INFO: Total idle time: 24.127683589000007
[1]<1> 13:45:36.704: (active,inactive) pixels processed: (8923948,95947)
114.005165 seconds (48.30 M allocations: 3.554 GiB, 1.54% gc time)

Constant

[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [82.4164, 80.7837, 82.298, 81.5449, 91.8007, 80.8242, 80.9523, 81.0272]
[1]<1> INFO: Batch 1 avg threads idle: 10%
[1]<1> INFO: Batch 1 - [4.26968, 4.39222, 4.30624, 10.968, 4.46464, 4.36652, 4.43842, 4.35103]
[1]<1> INFO: Batch 1 avg threads idle: 53%
[1]<1> INFO: Batch 1 - [3.69239, 3.74899, 4.00674, 3.72065, 3.76149, 3.75833, 3.85371, 5.58136]
[1]<1> INFO: Batch 1 avg threads idle: 28%
[1]<1> 14:23:29.16: (active,inactive) pixels processed: (15380206,563674)
132.618821 seconds (59.45 M allocations: 3.610 GiB, 1.79% gc time)

226.5, 226.625, 3.0, 3.125

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [62.555, 60.704, 61.5642, 61.1678, 61.9296, 60.7379, 61.1947, 60.7453]
[1]<1> INFO: Batch 1 avg threads idle: 2% (1.2301783913749995 / 62.554983885)
[1]<1> INFO: Batch 2 - [22.7043, 22.4418, 26.5636, 23.6258, 22.4239, 23.0652, 22.5344, 23.1717]
[1]<1> INFO: Batch 2 avg threads idle: 12% (3.2472487829999994 / 26.56357855)
[1]<1> INFO: Batch 1 - [7.13963, 2.20152, 2.15404, 2.24826, 2.1383, 2.13739, 2.21707, 2.23685]
[1]<1> INFO: Batch 1 avg threads idle: 61% (4.3304984788750005 / 7.139631808)
[1]<1> INFO: Batch 2 - [2.22321, 2.10545, 2.08467, 2.09797, 2.03988, 2.0844, 2.0672, 2.18643]
[1]<1> INFO: Batch 2 avg threads idle: 5% (0.11205953674999997 / 2.223209775)
[1]<1> INFO: Batch 1 - [2.04932, 2.07378, 2.05425, 1.99572, 2.1042, 2.06283, 2.0086, 2.04448]
[1]<1> INFO: Batch 1 avg threads idle: 3% (0.055051434124999826 / 2.104199106)
[1]<1> INFO: Batch 2 - [1.94024, 1.94868, 1.84555, 1.84402, 1.80236, 1.80778, 1.83883, 1.79082]
[1]<1> INFO: Batch 2 avg threads idle: 5% (0.09639532149999996 / 1.948678897)
[1]<1> INFO: Total idle time: 72.57145556499998
[1]<1> 13:48:11.989: (active,inactive) pixels processed: (15008794,503479)
131.984952 seconds (58.98 M allocations: 3.611 GiB, 1.78% gc time)

Constant

[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [82.4164, 80.7837, 82.298, 81.5449, 91.8007, 80.8242, 80.9523, 81.0272]
[1]<1> INFO: Batch 1 avg threads idle: 10%
[1]<1> INFO: Batch 1 - [4.26968, 4.39222, 4.30624, 10.968, 4.46464, 4.36652, 4.43842, 4.35103]
[1]<1> INFO: Batch 1 avg threads idle: 53%
[1]<1> INFO: Batch 1 - [3.69239, 3.74899, 4.00674, 3.72065, 3.76149, 3.75833, 3.85371, 5.58136]
[1]<1> INFO: Batch 1 avg threads idle: 28%
[1]<1> 14:23:29.16: (active,inactive) pixels processed: (15380206,563674)
132.618821 seconds (59.45 M allocations: 3.610 GiB, 1.79% gc time)

6.0, 6.125, -13.875, -13.75

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [56.2853, 56.5416, 56.7853, 56.4541, 56.6277, 56.3169, 57.1147, 56.3027]
[1]<1> INFO: Batch 1 avg threads idle: 1% (0.5611397045000013 / 57.11466876)
[1]<1> INFO: Batch 2 - [0.41917, 0.48143, 0.726374, 0.543214, 0.649233, 1.59521, 0.000107658, 0.681676]
[1]<1> INFO: Batch 2 avg threads idle: 60% (0.9581591611250001 / 1.59521127)
[1]<1> INFO: Batch 1 - [1.66742, 1.71257, 1.6835, 1.68338, 1.69295, 1.65961, 1.66428, 1.68197]
[1]<1> INFO: Batch 1 avg threads idle: 2% (0.03186161974999993 / 1.712572136)
[1]<1> INFO: Batch 2 - [1.7631e-5, 0.0623615, 0.0615055, 0.0649998, 0.0411647, 0.0446648, 0.0637264, 0.0611743]
[1]<1> INFO: Batch 2 avg threads idle: 23% (0.015047943375 / 0.064999769)
[1]<1> INFO: Batch 1 - [1.65561, 1.62641, 1.68077, 1.63561, 1.63905, 1.63155, 1.64759, 1.62392]
[1]<1> INFO: Batch 1 avg threads idle: 2% (0.038208950750000026 / 1.680772439)
[1]<1> INFO: Batch 2 - [0.0607345, 0.0598217, 0.0602053, 0.039219, 0.0582935, 2.7732e-5, 0.042103, 0.0579584]
[1]<1> INFO: Batch 2 avg threads idle: 22% (0.013439118749999998 / 0.06073451)
[1]<1> INFO: Total idle time: 12.942851986000012
[1]<1> 13:50:02.42: (active,inactive) pixels processed: (5556377,29386)
 89.669538 seconds (37.97 M allocations: 2.902 GiB, 2.24% gc time)

Constant

[1]<1> INFO: Done assigning sources to threads for processing
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [53.2234, 53.4707, 54.3479, 54.0878, 53.4181, 53.3721, 53.6616, 53.5435]
[1]<1> INFO: Batch 1 avg threads idle: 1%
[1]<1> INFO: Batch 1 - [1.47401, 1.47772, 1.53551, 1.4985, 1.48744, 1.48024, 2.12229, 1.50501]
[1]<1> INFO: Batch 1 avg threads idle: 26%
[1]<1> INFO: Batch 1 - [1.55299, 1.56477, 1.5723, 1.52525, 1.57112, 2.53405, 1.55099, 1.53884]
[1]<1> INFO: Batch 1 avg threads idle: 34%
[1]<1> 14:25:12.744: (active,inactive) pixels processed: (5555407,29191)
 81.483796 seconds (36.53 M allocations: 2.813 GiB, 2.45% gc time)

200.0, 200.125, 10.0, 10.125

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [65.5877, 62.6994, 62.3517, 63.7874, 62.5066, 62.6083, 62.9313, 62.3071]
[1]<1> INFO: Batch 1 avg threads idle: 4% (2.490278772750006 / 65.587718218)
[1]<1> INFO: Batch 2 - [19.0814, 18.9008, 18.6417, 19.4398, 19.3597, 19.7148, 18.7935, 19.6517]
[1]<1> INFO: Batch 2 avg threads idle: 3% (0.5168836161249981 / 19.714820586)
[1]<1> INFO: Batch 1 - [2.21987, 2.27699, 2.19899, 2.28122, 2.21095, 2.30785, 2.20802, 2.14607]
[1]<1> INFO: Batch 1 avg threads idle: 3% (0.07660574400000014 / 2.307851213)
[1]<1> INFO: Batch 2 - [1.58728, 1.54412, 1.56879, 1.58859, 1.69864, 1.59465, 1.63809, 1.67657]
[1]<1> INFO: Batch 2 avg threads idle: 5% (0.08655032587500006 / 1.69864089)
[1]<1> INFO: Batch 1 - [2.13742, 2.08342, 2.15073, 2.07218, 2.15183, 2.12615, 2.22068, 2.05539]
[1]<1> INFO: Batch 1 avg threads idle: 4% (0.09595597062500005 / 2.220681056)
[1]<1> INFO: Batch 2 - [1.53752, 1.59228, 1.61839, 1.50666, 1.53861, 1.50141, 1.54781, 1.49173]
[1]<1> INFO: Batch 2 avg threads idle: 5% (0.07658574637499996 / 1.61838836)
[1]<1> INFO: Total idle time: 26.742881406000038
[1]<1> 13:52:30.842: (active,inactive) pixels processed: (13505766,34908)
125.922355 seconds (59.69 M allocations: 3.897 GiB, 1.46% gc time)

Constant

[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [78.8138, 79.1887, 79.0962, 79.4395, 81.725, 80.8167, 78.7741, 82.6797]
[1]<1> INFO: Batch 1 avg threads idle: 3%
[1]<1> INFO: Batch 1 - [3.73674, 3.86074, 3.66881, 3.66104, 3.71193, 4.06571, 3.70091, 4.10662]
[1]<1> INFO: Batch 1 avg threads idle: 7%
[1]<1> INFO: Batch 1 - [3.80793, 3.45051, 3.47014, 3.46977, 3.49666, 3.82641, 3.51017, 3.55844]
[1]<1> INFO: Batch 1 avg threads idle: 7%
[1]<1> 14:27:33.093: (active,inactive) pixels processed: (13510402,35018)
116.496985 seconds (58.62 M allocations: 3.849 GiB, 1.81% gc time)

141.51, 141.76, 58.74, 58.99

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [166.684, 167.85, 166.689, 166.556, 166.636, 167.495, 166.129, 166.691]
[1]<1> INFO: Batch 1 avg threads idle: 1% (1.0087092680000005 / 167.850008963)
[1]<1> INFO: Batch 1 - [12.95, 11.5989, 12.9488, 11.6392, 11.6776, 11.6421, 13.1165, 11.629]
[1]<1> INFO: Batch 1 avg threads idle: 7% (0.9662699147499993 / 13.116534534)
[1]<1> INFO: Batch 1 - [11.0079, 10.9866, 10.9962, 10.9992, 10.999, 11.0338, 11.0589, 11.077]
[1]<1> INFO: Batch 1 avg threads idle: 1% (0.057170509624999255 / 11.07700027)
[1]<1> INFO: Total idle time: 16.257197538999993
[1]<1> 13:56:47.71: (active,inactive) pixels processed: (41393049,285253)
234.311401 seconds (132.22 M allocations: 7.522 GiB, 1.26% gc time)

Constant

[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [166.243, 167.375, 166.881, 166.292, 167.359, 166.755, 165.758, 166.142]
[1]<1> INFO: Batch 1 avg threads idle: 0%
[1]<1> INFO: Batch 1 - [13.6542, 13.6055, 13.7376, 13.6396, 13.5981, 13.5751, 13.5883, 13.6443]
[1]<1> INFO: Batch 1 avg threads idle: 1%
[1]<1> INFO: Batch 1 - [10.929, 10.9551, 11.039, 10.9082, 11.1382, 10.9665, 11.0374, 10.976]
[1]<1> INFO: Batch 1 avg threads idle: 1%
[1]<1> 14:31:37.647: (active,inactive) pixels processed: (41640950,343773)
223.618897 seconds (132.71 M allocations: 7.544 GiB, 1.49% gc time)

122.385, 122.51, 27.865, 27.99

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [119.121, 117.19, 118.225, 118.283, 119.501, 118.882, 119.465, 118.613]
[1]<1> INFO: Batch 1 avg threads idle: 1% (0.8411399916249973 / 119.501195407)
[1]<1> INFO: Batch 2 - [10.0146, 4.5668, 4.95268, 5.74462, 5.26087, 5.1965, 4.93608, 4.00343]
[1]<1> INFO: Batch 2 avg threads idle: 44% (4.4301823188750005 / 10.014632464)
[1]<1> INFO: Batch 1 - [8.14319, 8.23296, 10.2165, 8.47029, 10.4588, 8.16125, 8.00297, 10.7437]
[1]<1> INFO: Batch 1 avg threads idle: 16% (1.689991806 / 10.74368898)
[1]<1> INFO: Batch 2 - [0.47142, 0.424078, 0.421758, 0.503981, 0.299971, 0.386238, 0.436606, 0.545124]
[1]<1> INFO: Batch 2 avg threads idle: 20% (0.10897690825000003 / 0.545123793)
[1]<1> INFO: Batch 1 - [6.46686, 6.6036, 6.60363, 6.58202, 6.74195, 6.65979, 7.17486, 6.66117]
[1]<1> INFO: Batch 1 avg threads idle: 7% (0.48812733549999987 / 7.17486278)
[1]<1> INFO: Batch 2 - [0.484296, 0.78507, 0.399859, 0.391754, 0.393634, 0.350608, 0.477756, 0.371294]
[1]<1> INFO: Batch 2 avg threads idle: 42% (0.32828603349999996 / 0.785069889)
[1]<1> INFO: Total idle time: 63.09363514999998
[1]<1> 14:00:20.239: (active,inactive) pixels processed: (27799794,679676)
190.446911 seconds (101.27 M allocations: 5.747 GiB, 1.63% gc time)

Constant

[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [120.094, 126.052, 121.027, 121.751, 121.675, 122.449, 123.812, 122.035]
[1]<1> INFO: Batch 1 avg threads idle: 3%
[1]<1> INFO: Batch 1 - [8.63689, 8.60494, 8.77685, 8.6398, 8.6475, 8.64125, 10.0786, 9.7354]
[1]<1> INFO: Batch 1 avg threads idle: 11%
[1]<1> INFO: Batch 1 - [6.91529, 6.73749, 6.84932, 6.81562, 6.86577, 6.7938, 7.17856, 6.79601]
[1]<1> INFO: Batch 1 avg threads idle: 4%
[1]<1> 14:34:53.697: (active,inactive) pixels processed: (27664682,708166)
173.735256 seconds (101.10 M allocations: 5.748 GiB, 2.06% gc time)

327.51, 327.635, -3.01, -2.885

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing

[1]<1> INFO: Batch 1 - [86.859, 87.2472, 85.992, 85.4292, 86.3603, 85.2202, 86.6474, 85.2145]
[1]<1> INFO: Batch 1 avg threads idle: 1% (1.126022832499995 / 87.247239524)
[1]<1> INFO: Batch 2 - [35.4758, 34.8701, 36.7179, 35.169, 35.0006, 36.6606, 35.0877, 35.5966]
[1]<1> INFO: Batch 2 avg threads idle: 3% (1.1456441045000005 / 36.717927969)
[1]<1> INFO: Batch 1 - [5.11047, 5.20744, 5.10325, 5.17314, 5.1346, 5.14768, 5.21699, 5.16326]
[1]<1> INFO: Batch 1 avg threads idle: 1% (0.05988515024999996 / 5.216986143)
[1]<1> INFO: Batch 2 - [2.96034, 3.00282, 3.00961, 3.31572, 2.93065, 2.99717, 3.13542, 2.98243]
[1]<1> INFO: Batch 2 avg threads idle: 8% (0.273949647625 / 3.315719906)
[1]<1> INFO: Batch 1 - [4.20673, 4.19859, 4.27178, 4.22641, 4.18193, 4.24572, 4.28588, 4.22227]
[1]<1> INFO: Batch 1 avg threads idle: 1% (0.05596435700000024 / 4.28587697)
[1]<1> INFO: Batch 2 - [2.81457, 2.82477, 2.79399, 3.07212, 2.78059, 2.80808, 2.79776, 2.97153]
[1]<1> INFO: Batch 2 avg threads idle: 7% (0.21419679887500004 / 3.072123586)
[1]<1> INFO: Total idle time: 23.005303125999966
[1]<1> 14:03:34.405: (active,inactive) pixels processed: (27295830,625665)
173.002532 seconds (93.46 M allocations: 5.660 GiB, 1.63% gc time)

Constant

[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [123.439, 123.969, 121.144, 121.034, 121.934, 121.515, 122.001, 121.6]
[1]<1> INFO: Batch 1 avg threads idle: 2%
[1]<1> INFO: Batch 1 - [8.24348, 8.43454, 8.55559, 8.24985, 8.26625, 8.17231, 8.27387, 8.24118]
[1]<1> INFO: Batch 1 avg threads idle: 3%
[1]<1> INFO: Batch 1 - [7.58944, 7.3758, 7.65082, 7.64835, 7.41014, 7.63967, 7.43034, 7.37449]
[1]<1> INFO: Batch 1 avg threads idle: 2%
[1]<1> 14:38:00.643: (active,inactive) pixels processed: (27490752,710929)
166.094552 seconds (93.80 M allocations: 5.633 GiB, 1.81% gc time)

44.16625, 44.1975, 0.99, 1.02125

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [83.09, 62.4687, 44.5621, 66.7631, 46.3117, 45.8908, 64.2922, 49.3444]
[1]<1> INFO: Batch 1 avg threads idle: 30% (25.249643034124993 / 83.090028158)
[1]<1> INFO: Batch 2 - [33.3046, 18.8987, 20.3285, 18.1675, 22.5462, 20.6033, 14.4996, 18.1893]
[1]<1> INFO: Batch 2 avg threads idle: 37% (12.487363257374998 / 33.30455969)
[1]<1> INFO: Batch 1 - [7.68709, 2.2775, 3.13749, 1.04989, 1.93011, 2.96195, 0.814412, 1.6902]
[1]<1> INFO: Batch 1 avg threads idle: 65% (4.9935049385 / 7.687085932)
[1]<1> INFO: Batch 2 - [2.23279, 2.29489, 1.69873, 1.78037, 1.73072, 1.62492, 3.19802, 1.81843]
[1]<1> INFO: Batch 2 avg threads idle: 36% (1.15066087 / 3.198018782)
[1]<1> INFO: Batch 1 - [1.10832, 0.78563, 2.33028, 3.09897, 3.15762, 1.48801, 1.67622, 0.661792]
[1]<1> INFO: Batch 1 avg threads idle: 43% (1.369263658125 / 3.15761866)
[1]<1> INFO: Batch 2 - [1.69836, 2.30801, 1.81847, 3.19949, 2.22646, 1.61805, 1.68952, 1.73029]
[1]<1> INFO: Batch 2 avg threads idle: 36% (1.1634105062500002 / 3.199491951)
[1]<1> INFO: Total idle time: 371.310770115
[1]<1> 14:07:30.932: (active,inactive) pixels processed: (14791834,892081)
214.366941 seconds (64.01 M allocations: 4.291 GiB, 0.56% gc time)

Constant

[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [67.9939, 134.326, 65.0571, 66.4275, 79.5656, 85.6305, 71.0308, 64.6759]
[1]<1> INFO: Batch 1 avg threads idle: 41%
[1]<1> INFO: Batch 1 - [2.9886, 5.52163, 2.62946, 2.83671, 3.21323, 3.42048, 3.75113, 13.6434]
[1]<1> INFO: Batch 1 avg threads idle: 65%
[1]<1> INFO: Batch 1 - [3.24753, 3.26604, 7.08951, 4.01428, 2.91159, 3.405, 2.84131, 4.99111]
[1]<1> INFO: Batch 1 avg threads idle: 44%
[1]<1> 14:41:51.359: (active,inactive) pixels processed: (14791834,892081)
208.302020 seconds (64.51 M allocations: 4.306 GiB, 0.48% gc time)

85.51, 85.76, 28.49, 28.74

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [54.4237, 53.9057, 54.2921, 54.0676, 54.2805, 53.8932, 54.2418, 54.6244]
[1]<1> INFO: Batch 1 avg threads idle: 1% (0.4082432548749999 / 54.624379204)
[1]<1> INFO: Batch 2 - [9.407e-6, 1.7121e-5, 8.961e-6, 7.373e-6, 2.6916e-5, 5.995e-6, 2.5188e-5, 0.43662]
[1]<1> INFO: Batch 2 avg threads idle: 87% (0.3820299577499999 / 0.436620089)
[1]<1> INFO: Batch 1 - [1.03749, 1.0329, 1.04785, 1.05985, 1.04359, 2.31379, 1.08053, 1.05746]
[1]<1> INFO: Batch 1 avg threads idle: 48% (1.104604255125 / 2.313786914)
[1]<1> INFO: Batch 2 - [1.709e-5, 1.236e-5, 1.2633e-5, 1.422e-5, 6.54e-6, 0.0616188, 1.1682e-5, 1.0087e-5]
[1]<1> INFO: Batch 2 avg threads idle: 87% (0.053905898875 / 0.061618829)
[1]<1> INFO: Batch 1 - [0.95821, 0.934517, 0.924693, 0.930048, 0.964543, 0.940304, 0.941648, 0.976043]
[1]<1> INFO: Batch 1 avg threads idle: 3% (0.029792801375000025 / 0.976043376)
[1]<1> INFO: Batch 2 - [1.7206e-5, 1.3762e-5, 8.119e-6, 1.4964e-5, 6.54e-6, 0.0299661, 1.2124e-5, 1.0404e-5]
[1]<1> INFO: Batch 2 avg threads idle: 87% (0.0262099555 / 0.029966109)
[1]<1> INFO: Total idle time: 16.038288987999998
[1]<1> 14:09:17.541: (active,inactive) pixels processed: (3755246,113414)
 84.413895 seconds (37.00 M allocations: 2.804 GiB, 2.13% gc time)

Constant

[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [54.4078, 54.3975, 54.0424, 54.0178, 54.7792, 54.1679, 55.2344, 54.1632]
[1]<1> INFO: Batch 1 avg threads idle: 2%
[1]<1> INFO: Batch 1 - [1.17956, 1.20038, 1.15685, 1.13214, 1.15707, 1.13422, 1.1601, 3.32406]
[1]<1> INFO: Batch 1 avg threads idle: 57%
[1]<1> INFO: Batch 1 - [0.967222, 0.961592, 0.973388, 0.997332, 1.1259, 1.01101, 0.981752, 0.98337]
[1]<1> INFO: Batch 1 avg threads idle: 11%
[1]<1> 14:43:37.654: (active,inactive) pixels processed: (3818703,123924)
 83.471873 seconds (36.91 M allocations: 2.800 GiB, 2.57% gc time)

111.1875, 111.22, -14.0625, -14.03 (This was the box that originally triggered the discussion)

Auto

Partition cyclades auto batchsize...
[1]<1> Processing with dynamic connected components load balancing
[1]<1> INFO: Batch 1 - [49.3439, 48.9407, 52.7668, 47.8387, 46.2756, 46.3818, 50.5602, 52.3349]
[1]<1> INFO: Batch 1 avg threads idle: 7% (3.461514535624997 / 52.766848317)
[1]<1> INFO: Batch 2 - [12.1657, 9.20612, 7.18913, 8.71023, 8.5407, 11.1126, 8.68124, 11.2685]
[1]<1> INFO: Batch 2 avg threads idle: 21% (2.5564636978750004 / 12.165745974)
[1]<1> INFO: Batch 3 - [4.62941, 6.19663, 6.63504, 8.14304, 5.20088, 16.1447, 16.2192, 10.4044]
[1]<1> INFO: Batch 3 avg threads idle: 43% (7.022530954999999 / 16.219191773)
[1]<1> INFO: Batch 1 - [0.849308, 1.07229, 0.922506, 1.24186, 4.57341, 5.06963, 3.40124, 0.827647]
[1]<1> INFO: Batch 1 avg threads idle: 56% (2.824895903625 / 5.06963312)
[1]<1> INFO: Batch 2 - [2.02792, 1.15864, 1.41148, 2.32703, 1.23896, 2.65542, 4.83896, 1.75024]
[1]<1> INFO: Batch 2 avg threads idle: 55% (2.6628826412499995 / 4.838964394)
[1]<1> INFO: Batch 3 - [4.42037, 2.32421, 0.462527, 0.701052, 2.19179, 4.21366, 0.361551, 0.516813]
[1]<1> INFO: Batch 3 avg threads idle: 57% (2.5213707248749997 / 4.420366011)
[1]<1> INFO: Batch 1 - [1.52753, 1.48388, 0.990953, 0.762582, 0.742802, 0.546413, 0.775617, 1.06765]
[1]<1> INFO: Batch 1 avg threads idle: 35% (0.5403519414999999 / 1.527529722)
[1]<1> INFO: Batch 2 - [0.661068, 0.719738, 0.66943, 0.771404, 0.754844, 0.878169, 0.661495, 0.754177]
[1]<1> INFO: Batch 2 avg threads idle: 16% (0.14437831375 / 0.878168932)
[1]<1> INFO: Batch 3 - [0.462592, 1.09915, 0.328417, 0.338571, 0.359693, 0.795419, 1.38585, 1.07773]
[1]<1> INFO: Batch 3 avg threads idle: 47% (0.6549213888750001 / 1.385849534)
[1]<1> INFO: Total idle time: 179.11448081899994
[1]<1> 14:55:56.106: (active,inactive) pixels processed: (9895724,2871136)
122.836675 seconds (54.42 M allocations: 2.549 GiB, 1.85% gc time)

Constant

[1]<1> INFO: Batch 1 - [43.3756, 204.682, 42.8026, 45.0954, 42.055, 43.052, 44.0449, 52.009]
[1]<1> INFO: Batch 1 avg threads idle: 68%
[1]<1> INFO: Batch 1 - [0.118574, 0.123472, 0.15818, 1.30827, 0.0602595, 0.335063, 33.142, 0.186375]
[1]<1> INFO: Batch 1 avg threads idle: 87%
[1]<1> INFO: Batch 1 - [0.114834, 0.160334, 0.347057, 0.186547, 0.0961494, 1.23463, 0.123578, 17.8119]
[1]<1> INFO: Batch 1 avg threads idle: 86%
[1]<1> 14:52:02.677: (active,inactive) pixels processed: (9678076,2725171)
284.466482 seconds (55.39 M allocations: 2.596 GiB, 0.88% gc time)
agnusmaximus commented 7 years ago

Hmm seems like on average the auto batchsize is worse... Though on very extreme cases (the box previously raised) the auto batchsize does better