gcc -> clang

[yakra]

repo	head
DataProcessing	~2b6cf669565550c2ddd721445e0da7249c976b89~ d1f8f34d1b73668539fe0d7e19e7cd9d23213375
HighwayData	2ea2e8696d6a70be265c032e5469a58081e4eb31
UserData	08267032c56fc144dd47d6dadb3aeb3dbf4c7d68

[yakra]

siteupdateST

task	BT	lab1	lab5	lab2	lab3	lab4
Reading region, country, and continent descriptions	0.001 0.005	0.001 0.001	0 0	0.001 0	0.001 0.001	0.002 0.002
Reading systems list in	0.659 0.591	0.429 0.424	0.342 0.324	0.521 0.503	0.68 0.682	0.65 0.62
Finding all .wpt files.	0.485 0.473	0.225 0.219	0.247 0.24	0.272 0.27	0.365 0.369	0.381 0.376
Reading waypoints for all routes	13.985 13.428	6.741 6.902	7.581 6.983	8.851 8.955	11.836 12.01	11.776 10.944
Sorting waypoints in Quadtree	3.645 3.533	2.114 2.038	2.628 2.203	2.675 2.595	3.868 3.627	3.904 3.284
Finding unprocessed wpt files	0.012 0.011	0.005 0.006	0.01 0.01	0.008 0.009	0.01 0.01	0.01 0.011
Searching for near-miss points	3.371 3.222	1.886 1.89	2.347 2.08	2.485 2.442	3.235 3.113	3.517 3.113
Near-miss point log and tm-master.nmp file	0.629 0.606	0.324 0.323	0.399 0.374	0.407 0.404	0.577 0.577	0.588 0.556
Writing near-miss point merged wpt files	6.135 5.971	6.516 5.973	3.267 3.314	7.757 7.606	9.975 9.753	5.221 5.232
Concurrent segment detection	0.763 0.801	0.541 0.562	0.488 0.497	0.65 0.674	0.966 1.017	0.753 0.753
Creating label hashes and checking route integrity	1.28 1.186	0.699 0.687	0.744 0.664	0.869 0.85	1.248 1.151	1.118 1.001
Reading updates file	0.029 0.026	0.015 0.022	0.015 0.015	0.021 0.021	0.03 0.027	0.023 0.021
Reading systemupdates file	0 0.001	0.002 0.001	0 0.003	0.001 0.001	0.001 0.002	0 0.001
Processing traveler list files:	16.054 16.247	8.387 7.855	9.157 8.352	9.439 9.289	12.634 12.441	13.488 12.306
Processed [0-9]+ traveler list files. Sorting and numbering	0 0	0 0	0 0	0 0	0 0	0 0
Clearing route . label hash tables	0.202 0.18	0.098 0.097	0.112 0.094	0.12 0.111	0.185 0.173	0.165 0.145
Writing pointsinuse.log, unusedaltlabels.log, listnamesinuse.log and unusedaltroutenames.log	0.795 0.49	0.313 0.3	0.361 0.303	0.361 0.343	0.484 0.446	0.541 0.452
Augmenting travelers for detected concurrent segments	7.357 6.722	3.774 3.841	3.328 3.249	4.168 4.191	5.552 5.527	4.716 4.772
Performing per-route data checks and computing stats	7.013 7.138	3.539 3.358	4.387 3.888	4.35 4.122	5.871 5.419	6.673 6.061
Writing highway data stats log file	0.195 0.187	0.142 0.141	0.107 0.111	0.172 0.178	0.233 0.229	0.158 0.173
Creating per-traveler stats logs and augmenting data structure	38.238 38.523	27.282 25.807	27.206 27.29	32.042 30.303	40.631 38.52	38.64 38.625
Writing allbyregionactiveonly.csv	0.537 0.552	0.266 0.338	0.232 0.243	0.317 0.346	0.419 0.43	0.352 0.372
Writing allbyregionactivepreview.csv	0.744 0.76	0.366 0.375	0.321 0.335	0.445 0.446	0.584 0.592	0.48 0.508
Writing per-system stats csv files	0.64 0.64	0.335 0.342	0.3 0.307	0.414 0.416	0.545 0.552	0.465 0.466
Reading datacheckfps.csv	0.011 0.014	0.009 0.009	0.004 0.005	0.011 0.01	0.019 0.014	0.01 0.01
Setting up for graphs of highway data	0.755 0.71	0.495 0.478	0.599 0.541	0.589 0.557	0.816 0.771	0.896 0.811
Creating unique names and vertices	4.378 4	2.643 2.608	2.606 2.37	3.249 3.174	4.189 4.188	3.934 3.547
Creating edges	1.525 1.467	0.886 0.905	1.042 1.035	1.135 1.121	1.5 1.526	1.607 1.575
Compressing collapsed edges	1.144 1.073	0.66 0.648	0.835 0.747	0.82 0.781	1.057 1	1.243 1.116
Writing graph waypoint simplification log	0.148 0.146	0.094 0.096	0.088 0.086	0.112 0.114	0.15 0.146	0.132 0.132
Writing master TM graph files	6.577 6.7	5.793 5.883	3.944 3.932	7.156 7.32	9.026 9.464	5.912 5.892
Creating continent graphs	12.93 13.076	9.148 9.446	9.406 8.88	10.814 11.063	13.838 14.349	13.75 12.987
Creating multisystem graphs	3.104 2.788	1.959 2.013	1.833 1.791	2.255 2.28	2.853 2.94	2.685 2.622
Creating system data graphs	3.024 3.024	2.216 2.268	1.976 1.958	2.507 2.557	3.139 3.263	2.844 2.81
Creating country graphs	9.496 9.761	6.689 6.892	6.388 6.193	7.617 7.766	9.823 10.082	9.136 8.837
Creating multiregion graphs	3.166 3.033	2.127 2.18	2.035 1.971	2.319 2.351	3.027 3.069	2.736 2.651
Creating area data graphs	1.031 0.844	0.521 0.522	0.54 0.506	0.593 0.589	0.773 0.783	0.798 0.746
Creating regional data graphs	10.485 10.894	6.662 6.755	6.196 6.006	7.144 7.205	9.441 9.701	8.726 8.458
Clearing HighwayGraph contents from memory	1.232 1.143	0.698 0.676	0.883 0.776	0.885 0.818	1.182 1.091	1.319 1.172
Marking datacheck false positives	3.469 3.591	2.146 2.31	2.074 2.15	2.78 2.998	3.581 3.79	3.435 3.661
Writing log of unmatched datacheck FP entries	0.003 0.004	0.004 0.001	0.001 0.002	0.002 0.002	0.004 0.001	0.004 0.003
Writing datacheck.log	0.008 0.007	0.015 0.019	0.006 0.002	0.02 0.02	0.027 0.034	0.006 0.007
...continents, countries, regions	0.001 0	0 0	0 0	0.001 0.001	0 0	0.001 0.001
...systems	0 0.001	0.003 0.002	0.001 0.001	0.001 0.001	0.001 0.001	0.003 0
...routes	0.352 0.358	0.224 0.216	0.144 0.14	0.267 0.26	0.355 0.341	0.224 0.207
...connectedRoutes	0.187 0.179	0.124 0.125	0.101 0.095	0.153 0.156	0.194 0.192	0.143 0.142
...connectedRouteRoots	0.006 0.009	0.006 0.003	0.004 0.002	0.004 0.003	0.005 0.009	0.007 0.004
...waypoints	3.808 3.72	2.469 2.424	2.004 1.97	3.029 2.993	3.943 4.112	3.09 3.063
...segments	6.748 6.996	4.919 4.952	4.25 4.132	6.076 6.099	7.831 8.047	6.562 6.53
...clinched	1.661 1.657	1.062 1.076	1.064 1.084	1.266 1.296	1.633 1.69	1.546 1.567
...overallMileageByRegion	0.001 0.001	0.002 0.003	0 0.002	0 0	0.003 0.001	0.001 0.001
...systemMileageByRegion	0.002 0.18	0 0	0.003 0.002	0.001 0.003	0.001 0.005	0.002 0.007
...clinchedOverallMileageByRegion	0.041 0.042	0.023 0.022	0.022 0.022	0.029 0.029	0.04 0.037	0.035 0.033
...clinchedSystemMileageByRegion	0.013 0.01	0.012 0.012	0.008 0.008	0.014 0.017	0.021 0.026	0.011 0.01
...clinchedConnectedRoutes	0.082 0.088	0.055 0.055	0.046 0.047	0.067 0.062	0.095 0.101	0.069 0.07
...clinchedRoutes	0.098 0.102	0.064 0.062	0.055 0.055	0.078 0.078	0.097 0.102	0.082 0.083
...updates	0.011 0.01	0.007 0.008	0.006 0.007	0.01 0.01	0.013 0.015	0.009 0.01
...systemUpdates	0.758 0.756	0.419 0.414	0.494 0.491	0.488 0.484	0.626 0.627	0.696 0.692
...datacheckErrors	0.123 0.32	0.02 0.019	0.014 0.016	0.022 0.02	0.029 0.026	0.02 0.021
...graphs	0.107 0.038	0.003 0.012	0.001 0.002	0.005 0.015	0.007 0.018	0.002 0.002
Processed [0-9]+ highway systems	0.125 0.012	0.071 0.07	0.007 0.007	0.077 0.077	0.148 0.128	0.01 0.01
WaypointQuadtree contains [0-9]+ total nodes	0 0.006	0 0.002	0.002 0	0 0	0 0.002	0 0.003
Computing waypoint colocation stats, reporting all with 8 or more colocations:	1.197 1.05	0.701 0.679	0.685 0.646	0.854 0.813	1.111 1.059	0.989 0.93
Total run time:	180.58 179.107	116.951 115.369	112.948 108.562	138.797 137.22	180.529 179.425	166.297 160.22

Note: Total run times are all a few milliseconds above the sum of the data in the columns. This is because there are 3 instances of adjacent timestamps in siteupdate.log where nothing happens between them other than printing the first message to stdout and calculating+printing the 2nd timestamp. Which takes pretty close to 0s, but is nonetheless an opportunity for the clock to tick forward a notch. These missing milliseconds aren't reflected in the times listed for performing individual tasks.

[yakra]

multi-threaded tasks

• [x] Double-check whether all presets are up-to-date!

preset	task
ReadWpt	Reading waypoints for all routes
QtSort	Sorting waypoints in Quadtree
NmpSearch	Searching for near-miss points
NmpMerged	Writing near-miss point merged wpt files
RteInt	Creating label hashes and checking route integrity
ReadList	Processing traveler list files:
ConcAug	Augmenting travelers for detected concurrent segments
CompStatsR	Performing per-route data checks and computing stats
CompStatsT	Computing stats per traveler
UserLog	Creating per-traveler stats logs and augmenting data structure
HGVertex	Creating unique names and vertices
Subgraph	writing all graph files
Total	Total run time:

[yakra]

single-threaded tasks in multi-threaded siteupdate

top: g++ compile bottom: clang++ compile

task	BT	lab1	lab5	lab2	lab3	lab4
Reading region, country, and continent descriptions	0.0045 0.0025	0.00012 0.00012	0 0.00137	0.0005 0.00043	0.00329 0.00287	0.0035 0.00508
Reading systems list in	0.65475 0.58275	0.42137 0.40962	0.34737 0.352	0.52087 0.49656	0.69875 0.6692	0.62291 0.65425
Finding all .wpt files.	0.47425 0.47075	0.21737 0.22337	0.24375 0.24237	0.26668 0.26868	0.37875 0.37954	0.3792 0.39087
Finding unprocessed wpt files	0.0165 0.0355	0.017 0.01762	0.015 0.0135	0.01612 0.01606	0.02191 0.02054	0.01691 0.0152
Near-miss point log and tm-master.nmp file	0.69875 0.746	0.38112 0.37912	0.4365 0.41712	0.45762 0.45693	0.65908 0.64912	0.86087 0.81154
Concurrent segment detection	0.82 0.8005	0.58787 0.59725	0.50962 0.51562	0.69825 0.70906	1.32629 1.34654	1.09245 1.08495
Reading updates file	0.0305 0.029	0.02725 0.02712	0.01787 0.0165	0.03068 0.02987	0.04566 0.0445	0.02637 0.02504
Reading systemupdates file	0.00025 0.00075	0.00062 0.001	0.00012 0.00075	0.0005 0.00056	0.00108 0.00125	0.0015 0.0007
Processed [0-9]+ traveler list files. Sorting and numbering	0 0	0 0	0 0	0 0	0 0	0 0
Clearing route . label hash tables	0.23775 0.215	0.12812 0.1295	0.12125 0.1015	0.14593 0.14612	0.2855 0.2862	0.24275 0.20091
Writing pointsinuse.log, unusedaltlabels.log, listnamesinuse.log and unusedaltroutenames.log	0.5095 0.4445	0.2445 0.23962	0.32337 0.26737	0.30337 0.28518	0.45633 0.42862	0.6362 0.54654
Writing to concurrencies.log	0.49675 0.507	0.2685 0.27325	0.22937 0.22775	0.29393 0.29937	0.44775 0.44183	0.4262 0.42904
Writing highway data stats log file	0.22675 0.2805	0.17887 0.1745	0.11137 0.11337	0.18393 0.18668	0.27304 0.28383	0.19037 0.19275
Writing allbyregionactiveonly.csv	0.62175 0.62	0.35512 0.36012	0.25475 0.2615	0.37618 0.38156	0.52458 0.52554	0.44995 0.46675
Writing allbyregionactivepreview.csv	0.949 0.85025	0.41325 0.416	0.34437 0.35462	0.49768 0.50212	0.65575 0.66445	0.57816 0.5887
Writing per-system stats csv files	0.83075 0.8395	0.3805 0.384	0.31987 0.32237	0.45962 0.47218	0.67633 0.68466	0.50137 0.50679
Reading datacheckfps.csv	0.01275 0.01275	0.01062 0.01062	0.00587 0.00537	0.01212 0.01181	0.0255 0.02591	0.0097 0.0095
Start writing database file	15.3455 14.98675	17.01637 16.68187	8.67187 8.775	13.15818 12.9195	20.812 20.6022	14.2337 14.23575
Setting up for graphs of highway data	0.8435 0.8185	0.50512 0.47987	0.60537 0.55	0.60837 0.5705	1.24304 1.1957	1.2567 1.17916
Creating edges	2.02425 2.18025	0.95762 0.98187	1.18937 1.10112	1.22525 1.22087	2.71429 2.7002	2.54454 2.3855
Compressing collapsed edges	1.4325 1.37575	0.70087 0.68075	0.86512 0.777	0.85756 0.82268	2.4952 2.37179	2.16875 2.05029
Writing graph waypoint simplification log	0.2155 0.221	0.13225 0.272	0.115 0.11387	0.16493 0.16518	0.46016 0.43466	0.26679 0.26816
"Writing master TM graph files." (get master graph vertex & edge counts for terminal output before writing files)	0.0945 0.097	0.22962 0.23325	0.06437 0.0585	0.27431 0.27587	0.50566 0.4977	0.13362 0.1277
Clearing HighwayGraph contents from memory	1.61725 1.5755	0.81737 0.79237	0.96762 0.8855	1.00812 0.96743	1.74291 1.7462	1.76516 1.62666
Marking datacheck false positives	2.8895 3.03775	2.11687 2.18637	1.60325 1.47087	3.08431 3.25037	3.85641 4.06195	2.59837 2.37475
Writing log of unmatched datacheck FP entries	0.00075 0.001	0.00187 0.00262	0.0005 0.00037	0.00387 0.00262	0.004 0.00483	0.00112 0.00116
Writing datacheck.log	0.008 0.00825	0.00812 0.00762	0.00437 0.0045	0.01037 0.00768	0.01216 0.01254	0.01095 0.01045
Resume writing database file	0.02325 0.03175	0.01587 0.01575	0.0135 0.01212	0.01787 0.01656	0.02658 0.02508	0.0215 0.01945
Processed [0-9]+ highway systems	0.0175 0.018	0.072 0.07287	0.00762 0.00837	0.07981 0.08106	0.3915 0.41637	0.04683 0.05116
WaypointQuadtree contains [0-9]+ total nodes	0 0.00475	0 0.00262	0.00112 0.00087	0 0.00193	0.00012 0.00445	0.00195 0.00191
Computing waypoint colocation stats, reporting all with 8 or more colocations:	1.10425 1.06	0.71462 0.698	0.71487 0.679	0.87406 0.84618	1.47708 1.45595	1.47483 1.41904
SUM	32.20075 31.8535	26.92077 26.75066	18.1044 17.65017	25.63099 25.4116	42.2207 41.98422	32.56322 31.67975

Nota bene: Start writing database file looks OK, but may not be the most accurate, with HighwayGraph construction and a varying number of subgraph writes happening at the same time. See also siteupdateST. Not included: Setting up subgraphs. Can't be reliably measured; too much variability. Running 1 thread, this stat will include the time Writing master TM graph files. >1 threads, times sporadically include extra time (in some cases >10s) while partially writing the master graphs and while the DB is written in parallel. This step is not included in siteupdateST, and I won't hack a special version of MT siteupdate just to isolate this one datum. :grin:

[yakra]

---

[yakra]

siteupdateST

Redoing the table above, because lab4 was mistakenly run with different HighwayData & UserData heads from the other machines. The results don't change much, but still... top: g++ compile bottom: clang++ compile

task	BT	lab1	lab5	lab2	lab3	lab4
Reading region, country, and continent descriptions	0.001 0.005	0.001 0.001	0 0	0.001 0	0.001 0.001	0.007 0.005
Reading systems list in	0.659 0.591	0.429 0.424	0.342 0.324	0.521 0.503	0.68 0.682	0.575 0.625
Finding all .wpt files.	0.485 0.473	0.225 0.219	0.247 0.24	0.272 0.27	0.365 0.369	0.4 0.384
Reading waypoints for all routes	13.985 13.428	6.741 6.902	7.581 6.983	8.851 8.955	11.836 12.01	11.784 11.051
Sorting waypoints in Quadtree	3.645 3.533	2.114 2.038	2.628 2.203	2.675 2.595	3.868 3.627	3.953 3.332
Finding unprocessed wpt files	0.012 0.011	0.005 0.006	0.01 0.01	0.008 0.009	0.01 0.01	0.019 0.018
Searching for near-miss points	3.371 3.222	1.886 1.89	2.347 2.08	2.485 2.442	3.235 3.113	3.545 3.14
Near-miss point log and tm-master.nmp file	0.629 0.606	0.324 0.323	0.399 0.374	0.407 0.404	0.577 0.577	0.595 0.558
Writing near-miss point merged wpt files	6.135 5.971	6.516 5.973	3.267 3.314	7.757 7.606	9.975 9.753	5.181 5.186
Concurrent segment detection	0.763 0.801	0.541 0.562	0.488 0.497	0.65 0.674	0.966 1.017	0.814 0.756
Creating label hashes and checking route integrity	1.28 1.186	0.699 0.687	0.744 0.664	0.869 0.85	1.248 1.151	1.167 0.997
Reading updates file	0.029 0.026	0.015 0.022	0.015 0.015	0.021 0.021	0.03 0.027	0.022 0.021
Reading systemupdates file	0 0.001	0.002 0.001	0 0.003	0.001 0.001	0.001 0.002	0 0
Processing traveler list files:	16.054 16.247	8.387 7.855	9.157 8.352	9.439 9.289	12.634 12.441	14.084 13.225
Processed [0-9]+ traveler list files. Sorting and numbering	0 0	0 0	0 0	0 0	0 0	0 0
Clearing route . label hash tables	0.202 0.18	0.098 0.097	0.112 0.094	0.12 0.111	0.185 0.173	0.185 0.17
Writing pointsinuse.log, unusedaltlabels.log, listnamesinuse.log and unusedaltroutenames.log	0.795 0.49	0.313 0.3	0.361 0.303	0.361 0.343	0.484 0.446	0.564 0.46
Augmenting travelers for detected concurrent segments	7.357 6.722	3.774 3.841	3.328 3.249	4.168 4.191	5.552 5.527	5.067 4.837
Performing per-route data checks and computing stats	7.013 7.138	3.539 3.358	4.387 3.888	4.35 4.122	5.871 5.419	6.973 6.42
Writing highway data stats log file	0.195 0.187	0.142 0.141	0.107 0.111	0.172 0.178	0.233 0.229	0.17 0.185
Creating per-traveler stats logs and augmenting data structure	38.238 38.523	27.282 25.807	27.206 27.29	32.042 30.303	40.631 38.52	40.219 40.807
Writing allbyregionactiveonly.csv	0.537 0.552	0.266 0.338	0.232 0.243	0.317 0.346	0.419 0.43	0.352 0.367
Writing allbyregionactivepreview.csv	0.744 0.76	0.366 0.375	0.321 0.335	0.445 0.446	0.584 0.592	0.483 0.507
Writing per-system stats csv files	0.64 0.64	0.335 0.342	0.3 0.307	0.414 0.416	0.545 0.552	0.46 0.472
Reading datacheckfps.csv	0.011 0.014	0.009 0.009	0.004 0.005	0.011 0.01	0.019 0.014	0.012 0.01
Setting up for graphs of highway data	0.755 0.71	0.495 0.478	0.599 0.541	0.589 0.557	0.816 0.771	0.936 0.859
Creating unique names and vertices	4.378 4	2.643 2.608	2.606 2.37	3.249 3.174	4.189 4.188	3.929 3.544
Creating edges	1.525 1.467	0.886 0.905	1.042 1.035	1.135 1.121	1.5 1.526	1.61 1.611
Compressing collapsed edges	1.144 1.073	0.66 0.648	0.835 0.747	0.82 0.781	1.057 1	1.255 1.128
Writing graph waypoint simplification log	0.148 0.146	0.094 0.096	0.088 0.086	0.112 0.114	0.15 0.146	0.137 0.133
Writing master TM graph files	6.577 6.7	5.793 5.883	3.944 3.932	7.156 7.32	9.026 9.464	5.967 5.939
Creating continent graphs	12.93 13.076	9.148 9.446	9.406 8.88	10.814 11.063	13.838 14.349	13.845 13.057
Creating multisystem graphs	3.104 2.788	1.959 2.013	1.833 1.791	2.255 2.28	2.853 2.94	2.635 2.568
Creating system data graphs	3.024 3.024	2.216 2.268	1.976 1.958	2.507 2.557	3.139 3.263	2.817 2.786
Creating country graphs	9.496 9.761	6.689 6.892	6.388 6.193	7.617 7.766	9.823 10.082	9.198 8.88
Creating multiregion graphs	3.166 3.033	2.127 2.18	2.035 1.971	2.319 2.351	3.027 3.069	2.889 2.793
Creating area data graphs	1.031 0.844	0.521 0.522	0.54 0.506	0.593 0.589	0.773 0.783	0.797 0.746
Creating regional data graphs	10.485 10.894	6.662 6.755	6.196 6.006	7.144 7.205	9.441 9.701	8.759 8.467
Clearing HighwayGraph contents from memory	1.232 1.143	0.698 0.676	0.883 0.776	0.885 0.818	1.182 1.091	1.335 1.192
Marking datacheck false positives	3.469 3.591	2.146 2.31	2.074 2.15	2.78 2.998	3.581 3.79	3.287 3.467
Writing log of unmatched datacheck FP entries	0.003 0.004	0.004 0.001	0.001 0.002	0.002 0.002	0.004 0.001	0.003 0.004
Writing datacheck.log	0.008 0.007	0.015 0.019	0.006 0.002	0.02 0.02	0.027 0.034	0.007 0.005
...continents, countries, regions	0.001 0	0 0	0 0	0.001 0.001	0 0	0 0.001
...systems	0 0.001	0.003 0.002	0.001 0.001	0.001 0.001	0.001 0.001	0 0.002
...routes	0.352 0.358	0.224 0.216	0.144 0.14	0.267 0.26	0.355 0.341	0.231 0.21
...connectedRoutes	0.187 0.179	0.124 0.125	0.101 0.095	0.153 0.156	0.194 0.192	0.15 0.143
...connectedRouteRoots	0.006 0.009	0.006 0.003	0.004 0.002	0.004 0.003	0.005 0.009	0.005 0.007
...waypoints	3.808 3.72	2.469 2.424	2.004 1.97	3.029 2.993	3.943 4.112	3.128 3.063
...segments	6.748 6.996	4.919 4.952	4.25 4.132	6.076 6.099	7.831 8.047	6.647 6.626
...clinched	1.661 1.657	1.062 1.076	1.064 1.084	1.266 1.296	1.633 1.69	1.573 1.604
...overallMileageByRegion	0.001 0.001	0.002 0.003	0 0.002	0 0	0.003 0.001	0.001 0.001
...systemMileageByRegion	0.002 0.18	0 0	0.003 0.002	0.001 0.003	0.001 0.005	0.003 0.002
...clinchedOverallMileageByRegion	0.041 0.042	0.023 0.022	0.022 0.022	0.029 0.029	0.04 0.037	0.034 0.035
...clinchedSystemMileageByRegion	0.013 0.01	0.012 0.012	0.008 0.008	0.014 0.017	0.021 0.026	0.012 0.012
...clinchedConnectedRoutes	0.082 0.088	0.055 0.055	0.046 0.047	0.067 0.062	0.095 0.101	0.071 0.071
...clinchedRoutes	0.098 0.102	0.064 0.062	0.055 0.055	0.078 0.078	0.097 0.102	0.084 0.088
...updates	0.011 0.01	0.007 0.008	0.006 0.007	0.01 0.01	0.013 0.015	0.007 0.01
...systemUpdates	0.758 0.756	0.419 0.414	0.494 0.491	0.488 0.484	0.626 0.627	0.701 0.694
...datacheckErrors	0.123 0.32	0.02 0.019	0.014 0.016	0.022 0.02	0.029 0.026	0.022 0.02
...graphs	0.107 0.038	0.003 0.012	0.001 0.002	0.005 0.015	0.007 0.018	0.002 0.001
Processed [0-9]+ highway systems	0.125 0.012	0.071 0.07	0.007 0.007	0.077 0.077	0.148 0.128	0.01 0.011
WaypointQuadtree contains [0-9]+ total nodes	0 0.006	0 0.002	0.002 0	0 0	0 0.002	0.001 0
Computing waypoint colocation stats, reporting all with 8 or more colocations:	1.197 1.05	0.701 0.679	0.685 0.646	0.854 0.813	1.111 1.059	0.998 0.945
Total run time:	180.58 179.107	116.951 115.369	112.948 108.562	138.797 137.22	180.529 179.425	169.719 164.259

Note: Total run times are all a few milliseconds above the sum of the data in the columns. This is because there are 3 instances of adjacent timestamps in siteupdate.log where nothing happens between them other than printing the first message to stdout and calculating+printing the 2nd timestamp. Which takes pretty close to 0s, but is nonetheless an opportunity for the clock to tick forward a notch. These missing milliseconds aren't reflected in the times listed for performing individual tasks.

[yakra]

single-threaded tasks in multi-threaded siteupdate

Redoing the table above, because lab4 was mistakenly run with different HighwayData & UserData heads from the other machines. The results don't change much, but still... top: g++ compile bottom: clang++ compile

task	BT	lab1	lab5	lab2	lab3	lab4
Reading region, country, and continent descriptions	0.0045 0.0025	0.00012 0.00012	0 0.00137	0.0005 0.00043	0.00329 0.00287	0.00395 0.00508
Reading systems list in	0.65475 0.58275	0.42137 0.40962	0.34737 0.352	0.52087 0.49656	0.69875 0.6692	0.62112 0.62383
Finding all .wpt files.	0.47425 0.47075	0.21737 0.22337	0.24375 0.24237	0.26668 0.26868	0.37875 0.37954	0.38754 0.38416
Finding unprocessed wpt files	0.0165 0.0355	0.017 0.01762	0.015 0.0135	0.01612 0.01606	0.02191 0.02054	0.01995 0.01962
Near-miss point log and tm-master.nmp file	0.69875 0.746	0.38112 0.37912	0.4365 0.41712	0.45762 0.45693	0.65908 0.64912	0.86716 0.82237
Concurrent segment detection	0.82 0.8005	0.58787 0.59725	0.50962 0.51562	0.69825 0.70906	1.32629 1.34654	1.09787 1.10304
Reading updates file	0.0305 0.029	0.02725 0.02712	0.01787 0.0165	0.03068 0.02987	0.04566 0.0445	0.0275 0.02504
Reading systemupdates file	0.00025 0.00075	0.00062 0.001	0.00012 0.00075	0.0005 0.00056	0.00108 0.00125	0.00033 0.00045
Processed [0-9]+ traveler list files. Sorting and numbering	0 0	0 0	0 0	0 0	0 0	0.00095 0.00095
Clearing route . label hash tables	0.23775 0.215	0.12812 0.1295	0.12125 0.1015	0.14593 0.14612	0.2855 0.2862	0.252 0.21566
Writing pointsinuse.log, unusedaltlabels.log, listnamesinuse.log and unusedaltroutenames.log	0.5095 0.4445	0.2445 0.23962	0.32337 0.26737	0.30337 0.28518	0.45633 0.42862	0.64683 0.56795
Writing to concurrencies.log	0.49675 0.507	0.2685 0.27325	0.22937 0.22775	0.29393 0.29937	0.44775 0.44183	0.43729 0.43525
Writing highway data stats log file	0.22675 0.2805	0.17887 0.1745	0.11137 0.11337	0.18393 0.18668	0.27304 0.28383	0.19404 0.19833
Writing allbyregionactiveonly.csv	0.62175 0.62	0.35512 0.36012	0.25475 0.2615	0.37618 0.38156	0.52458 0.52554	0.4522 0.46412
Writing allbyregionactivepreview.csv	0.949 0.85025	0.41325 0.416	0.34437 0.35462	0.49768 0.50212	0.65575 0.66445	0.58054 0.59012
Writing per-system stats csv files	0.83075 0.8395	0.3805 0.384	0.31987 0.32237	0.45962 0.47218	0.67633 0.68466	0.50604 0.51187
Reading datacheckfps.csv	0.01275 0.01275	0.01062 0.01062	0.00587 0.00537	0.01212 0.01181	0.0255 0.02591	0.0095 0.00954
Setting up for graphs of highway data	0.8435 0.8185	0.50512 0.47987	0.60537 0.55	0.60837 0.5705	1.24304 1.1957	1.25741 1.16566
Creating edges	2.02425 2.18025	0.95762 0.98187	1.18937 1.10112	1.22525 1.22087	2.71429 2.7002	2.60316 2.41283
Compressing collapsed edges	1.4325 1.37575	0.70087 0.68075	0.86512 0.777	0.85756 0.82268	2.4952 2.37179	2.27216 2.0362
Writing graph waypoint simplification log	0.2155 0.221	0.13225 0.272	0.115 0.11387	0.16493 0.16518	0.46016 0.43466	0.27691 0.26941
"Writing master TM graph files." (get master graph vertex & edge counts for terminal output before writing files)	0.0945 0.097	0.22962 0.23325	0.06437 0.0585	0.27431 0.27587	0.50566 0.4977	0.13741 0.12683
Setting up subgraphs	1.91275 2.06375	2.63525 3.38212	0.62262 0.61375	0.52193 0.53518	0.6352 0.65616	0.5262 0.5292
Clearing HighwayGraph contents from memory	1.61725 1.5755	0.81737 0.79237	0.96762 0.8855	1.00812 0.96743	1.74291 1.7462	1.768 1.634
Marking datacheck false positives	2.8895 3.03775	2.11687 2.18637	1.60325 1.47087	3.08431 3.25037	3.85641 4.06195	2.49112 2.3195
Writing log of unmatched datacheck FP entries	0.00075 0.001	0.00187 0.00262	0.0005 0.00037	0.00387 0.00262	0.004 0.00483	0.00091 0.00087
Writing datacheck.log	0.008 0.00825	0.00812 0.00762	0.00437 0.0045	0.01037 0.00768	0.01216 0.01254	0.01004 0.01066
Resume writing database file	0.02325 0.03175	0.01587 0.01575	0.0135 0.01212	0.01787 0.01656	0.02658 0.02508	0.01983 0.01983
Processed [0-9]+ highway systems	0.0175 0.01325	0.072 0.07025	0.0065 0.0075	0.07981 0.07912	0.39137 0.41191	0.04495 0.048
WaypointQuadtree contains [0-9]+ total nodes	0 0.00475	0 0.00262	0.00112 0.00087	0 0.00193	0.00012 0.00445	0.00166 0.00141
Computing waypoint colocation stats, reporting all with 8 or more colocations:	1.10425 1.06	0.71462 0.698	0.71487 0.679	0.87406 0.84618	1.47708 1.45595	1.46283 1.4342
SUM	18.768 18.92575	12.53965 13.44829	10.05403 9.48805	12.99474 13.02534	22.04377 22.03372	18.9774 17.98598
Start writing database file	15.3455 14.98675	17.01637 16.68187	8.67187 8.775	13.15818 12.9195	20.812 20.6022	14.56658 14.46587

Nota bene:

• Setting up subgraphs can't be reliably measured; too much variability. Running "1" thread (this is the one place siteupdate briefly runs numthreads+1 threads), this stat will include the time Writing master TM graph files. >1 threads, times sporadically include extra time (in some cases >10s) while partially writing the master graphs and while the DB is written in parallel. This step is not included in siteupdateST, and I won't hack a special version of MT siteupdate just to isolate this one datum. :grin: Nonetheless, it's included in the table because it's part of the total time spent on single-threaded tasks.
• Start writing database file looks OK, but may not be the most accurate, with HighwayGraph construction and a varying number of subgraph writes happening at the same time. See also siteupdateST, where times are broken down into indidivual tables. This task includes all but the last two.
• Processed [0-9]+ highway systems time has changed slightly for clang on all machines vs the earlier table. I don't have any explanation for how/why this happened.

[yakra]

Redoing the charts, because lab4 was mistakenly run with different HighwayData & UserData heads from the other machines.

---