icecc / icecream

Distributed compiler with a central scheduler to share build load
GNU General Public License v2.0
1.58k stars 248 forks source link

Selection of compile server not optimal #593

Closed jedelbo closed 2 years ago

jedelbo commented 2 years ago

A have a simple setup with two compiler servers, and after upgrading to v1.3 I find that jobs are primarily scheduled on one server.

It seems to be due to the face that when client_count is greater than maxJobs(), the client gets penalized, resulting in a situation where it will never get any jobs. In the trace file below you can see that jed-dev-01 never wins a job after job 17.

I am not sure where the problem lies. What does client_count represent? In this case it grows to 10,

[297591] 2022-01-20 11:43:16: ICECREAM scheduler 1.3.90 starting up, port 8765
[297591] 2022-01-20 11:43:16: scheduler ready
[297591] 2022-01-20 11:43:16: ignoring localhost lo for broadcast
[297591] 2022-01-20 11:43:16: broadcast wlp0s20f3 192.168.1.255
[297591] 2022-01-20 11:43:16: broadcast docker0 172.17.255.255
[297591] 2022-01-20 11:43:16: ignoring localhost lo for broadcast
[297591] 2022-01-20 11:43:16: broadcast wlp0s20f3 192.168.1.255
[297591] 2022-01-20 11:43:16: broadcast docker0 172.17.255.255
[297591] 2022-01-20 11:43:16: Received scheduler announcement from 192.168.1.15:51849 (version 42, netname ICECREAM)
[297591] 2022-01-20 11:43:16: Received scheduler announcement from 172.17.0.1:51849 (version 42, netname ICECREAM)
[297591] 2022-01-20 11:43:17: broadcast from 192.168.1.15:57618 (version 42)
[297591] 2022-01-20 11:43:17: broadcast from 172.17.0.1:57618 (version 42)
[297591] 2022-01-20 11:43:19: broadcast from 192.168.1.3:58655 (version 42)
[297591] 2022-01-20 11:43:20: accepted 192.168.1.15
[297591] 2022-01-20 11:43:20: login jed-dev-01 protocol version: 42 features: env_xz env_zstd []
[297591] 2022-01-20 11:43:23: accepted 192.168.1.3
[297591] 2022-01-20 11:43:23: login HP-Z800 protocol version: 42 features: env_xz env_zstd [c161c43a4b8feaa9487be23f4e567534(x86_64), 75b761cfe4904e62f1715ab5b1a1ba9a(x86_64), ]
[297591] 2022-01-20 11:43:33: NEW 1 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] sync/noinst/changeset_index.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 1 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly selected HP-Z800 load: 122 can install: x86_64
[297591] 2022-01-20 11:43:33: put 1 in joblist of HP-Z800
[297591] 2022-01-20 11:43:33: NEW 2 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] sync/noinst/compact_changesets.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 2 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly selected jed-dev-01 load: 449 can install: x86_64
[297591] 2022-01-20 11:43:33: put 2 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:33: BEGIN: 2 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:33: NEW 3 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] sync/noinst/client_history_impl.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 3 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly selected jed-dev-01 load: 449 can install: x86_64
[297591] 2022-01-20 11:43:33: put 3 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:33: BEGIN: 3 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:33: BEGIN: 1 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:33: NEW 4 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] sync/noinst/client_reset.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 4 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly selected HP-Z800 load: 122 can install: x86_64
[297591] 2022-01-20 11:43:33: put 4 in joblist of HP-Z800
[297591] 2022-01-20 11:43:33: BEGIN: 4 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:33: NEW 5 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] sync/noinst/protocol_codec.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 5 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly selected jed-dev-01 load: 449 can install: x86_64
[297591] 2022-01-20 11:43:33: put 5 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:33: BEGIN: 5 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:33: NEW 6 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] sync/noinst/client_reset_operation.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 6 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly selected jed-dev-01 load: 449 can install: x86_64
[297591] 2022-01-20 11:43:33: put 6 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:33: BEGIN: 6 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:33: NEW 7 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/sync/changeset_parser.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 7 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly selected jed-dev-01 load: 449 can install: x86_64
[297591] 2022-01-20 11:43:33: put 7 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:33: BEGIN: 7 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:33: NEW 8 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/sync/changeset_encoder.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 8 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly selected HP-Z800 load: 122 can install: x86_64
[297591] 2022-01-20 11:43:33: put 8 in joblist of HP-Z800
[297591] 2022-01-20 11:43:33: BEGIN: 8 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:33: NEW 9 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] sync/noinst/client_impl_base.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 9 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly selected HP-Z800 load: 122 can install: x86_64
[297591] 2022-01-20 11:43:33: put 9 in joblist of HP-Z800
[297591] 2022-01-20 11:43:33: BEGIN: 9 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:35: NEW 10 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/parser/driver.cpp C++
[297591] 2022-01-20 11:43:35: pick_server 10 x86_64
[297591] 2022-01-20 11:43:35: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:35: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:35: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:35: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:35: no job stats - returning randomly selected HP-Z800 load: 122 can install: x86_64
[297591] 2022-01-20 11:43:35: put 10 in joblist of HP-Z800
[297591] 2022-01-20 11:43:35: BEGIN: 10 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:36: END 6 status=0 in=0(0%) out=2594472(0%) real=2610 user=2435 sys=139 pfaults=91581 server=jed-dev-01
[297591] 2022-01-20 11:43:36: add_job_stats C++ 20 2435 10000 720686 2594472 jed-dev-01 1065.49 1183.88
[297591] 2022-01-20 11:43:36: NEW 11 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/parser/keypath_mapping.cpp C++
[297591] 2022-01-20 11:43:36: pick_server 11 x86_64
[297591] 2022-01-20 11:43:36: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:36: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:36: jed-dev-01 compiled 1 got now: 4 speed: 88.7909 compile time 2435 produced code 720686 client count: 10
[297591] 2022-01-20 11:43:36: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:36: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:36: HP-Z800 compiled 0 got now: 5 speed: 0 compile time 0 produced code 0 client count: 5
[297591] 2022-01-20 11:43:36: penalizing local build for job 11
[297591] 2022-01-20 11:43:36: taking best installed jed-dev-01 88.7909
[297591] 2022-01-20 11:43:36: put 11 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:36: BEGIN: 11 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:36: END 2 status=0 in=0(0%) out=1904304(0%) real=3119 user=2916 sys=186 pfaults=88784 server=jed-dev-01
[297591] 2022-01-20 11:43:36: add_job_stats C++ 20 2916 10000 528973 1904304 jed-dev-01 653.054 817.381
[297591] 2022-01-20 11:43:37: NEW 12 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] parser/generated/query_flex.cpp C++
[297591] 2022-01-20 11:43:37: pick_server 12 x86_64
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 compiled 2 got now: 4 speed: 61.3036 compile time 5351 produced code 1249659 client count: 10
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:37: HP-Z800 compiled 0 got now: 5 speed: 0 compile time 0 produced code 0 client count: 5
[297591] 2022-01-20 11:43:37: penalizing local build for job 12
[297591] 2022-01-20 11:43:37: taking best installed jed-dev-01 61.3036
[297591] 2022-01-20 11:43:37: put 12 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:37: BEGIN: 12 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:37: END 5 status=0 in=0(0%) out=2070328(0%) real=3325 user=3115 sys=155 pfaults=91109 server=jed-dev-01
[297591] 2022-01-20 11:43:37: add_job_stats C++ 21 3115 10000 575091 2070328 jed-dev-01 664.632 646.616
[297591] 2022-01-20 11:43:37: NEW 13 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] parser/generated/query_bison.cpp C++
[297591] 2022-01-20 11:43:37: pick_server 13 x86_64
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 compiled 3 got now: 4 speed: 48.4962 compile time 8466 produced code 1824750 client count: 10
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:37: HP-Z800 compiled 0 got now: 5 speed: 0 compile time 0 produced code 0 client count: 5
[297591] 2022-01-20 11:43:37: penalizing local build for job 13
[297591] 2022-01-20 11:43:37: taking best installed jed-dev-01 48.4962
[297591] 2022-01-20 11:43:37: put 13 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:37: BEGIN: 13 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:37: END 7 status=0 in=0(0%) out=5887488(0%) real=3616 user=3422 sys=183 pfaults=107904 server=jed-dev-01
[297591] 2022-01-20 11:43:37: add_job_stats C++ 21 3422 10000 1635413 5887488 jed-dev-01 1720.48 727.659
[297591] 2022-01-20 11:43:37: NEW 14 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/collection.cpp C++
[297591] 2022-01-20 11:43:37: pick_server 14 x86_64
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 compiled 4 got now: 4 speed: 54.5744 compile time 11888 produced code 3460163 client count: 10
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:37: HP-Z800 compiled 0 got now: 5 speed: 0 compile time 0 produced code 0 client count: 5
[297591] 2022-01-20 11:43:37: penalizing local build for job 14
[297591] 2022-01-20 11:43:37: taking best installed jed-dev-01 54.5744
[297591] 2022-01-20 11:43:37: put 14 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:37: BEGIN: 14 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:38: END 3 status=0 in=0(0%) out=5847544(0%) real=4263 user=4019 sys=192 pfaults=131870 server=jed-dev-01
[297591] 2022-01-20 11:43:38: add_job_stats C++ 22 4019 10000 1624317 5847544 jed-dev-01 1454.97 639.276
[297591] 2022-01-20 11:43:38: END 11 status=0 in=0(0%) out=1854024(0%) real=1515 user=1378 sys=128 pfaults=57483 server=jed-dev-01
[297591] 2022-01-20 11:43:38: add_job_stats C++ 22 1378 10000 515006 1854024 jed-dev-01 1345.45 485.926
[297591] 2022-01-20 11:43:38: NEW 15 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/dictionary.cpp C++
[297591] 2022-01-20 11:43:38: pick_server 15 x86_64
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 compiled 6 got now: 3 speed: 39.4815 compile time 17285 produced code 5599486 client count: 9
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:38: HP-Z800 compiled 0 got now: 5 speed: 0 compile time 0 produced code 0 client count: 5
[297591] 2022-01-20 11:43:38: penalizing local build for job 15
[297591] 2022-01-20 11:43:38: taking best installed jed-dev-01 39.4815
[297591] 2022-01-20 11:43:38: put 15 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:38: BEGIN: 15 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:38: NEW 16 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/list.cpp C++
[297591] 2022-01-20 11:43:38: pick_server 16 x86_64
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 compiled 6 got now: 4 speed: 36.4444 compile time 17285 produced code 5599486 client count: 10
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:38: HP-Z800 compiled 0 got now: 5 speed: 0 compile time 0 produced code 0 client count: 0
[297591] 2022-01-20 11:43:38: penalizing local build for job 16
[297591] 2022-01-20 11:43:38: taking best installed jed-dev-01 36.4444
[297591] 2022-01-20 11:43:38: put 16 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:38: BEGIN: 16 client=jed-dev-01(x86_64) server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:38: END 8 status=0 in=4645457(20%) out=3208768(100%) real=4625 user=4221 sys=231 pfaults=94615 server=HP-Z800
[297591] 2022-01-20 11:43:38: add_job_stats C++ 22 4221 10000 891324 3208768 HP-Z800 760.191 844.657
[297591] 2022-01-20 11:43:38: NEW 17 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/object.cpp C++
[297591] 2022-01-20 11:43:38: pick_server 17 x86_64
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 compiled 6 got now: 5 speed: 33.4074 compile time 17285 produced code 5599486 client count: 10
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:38: HP-Z800 compiled 1 got now: 4 speed: 553.567 compile time 4221 produced code 891324 client count: 5
[297591] 2022-01-20 11:43:38: penalizing local build for job 17
[297591] 2022-01-20 11:43:38: taking best installed HP-Z800 553.567
[297591] 2022-01-20 11:43:38: put 17 in joblist of HP-Z800
[297591] 2022-01-20 11:43:38: BEGIN: 17 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:40: END 12 status=0 in=0(0%) out=3287496(0%) real=3042 user=2890 sys=147 pfaults=102761 server=jed-dev-01
[297591] 2022-01-20 11:43:40: add_job_stats C++ 24 2890 10000 913193 3287496 jed-dev-01 1137.54 322.809
[297591] 2022-01-20 11:43:40: END 4 status=0 in=5147606(20%) out=4320232(100%) real=6418 user=5851 sys=335 pfaults=132809 server=HP-Z800
[297591] 2022-01-20 11:43:40: add_job_stats C++ 24 5851 10000 1200064 4320232 HP-Z800 738.375 726.753
[297591] 2022-01-20 11:43:40: NEW 18 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/object_schema.cpp C++
[297591] 2022-01-20 11:43:40: pick_server 18 x86_64
[297591] 2022-01-20 11:43:40: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:40: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:40: jed-dev-01 compiled 7 got now: 4 speed: 24.2107 compile time 20175 produced code 6512679 client count: 10
[297591] 2022-01-20 11:43:40: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:40: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:40: HP-Z800 compiled 2 got now: 4 speed: 476.296 compile time 10072 produced code 2091388 client count: 5
[297591] 2022-01-20 11:43:40: penalizing local build for job 18
[297591] 2022-01-20 11:43:40: taking best installed HP-Z800 476.296
[297591] 2022-01-20 11:43:40: put 18 in joblist of HP-Z800
[297591] 2022-01-20 11:43:40: BEGIN: 18 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:40: NEW 19 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/object_store.cpp C++
[297591] 2022-01-20 11:43:40: pick_server 19 x86_64
[297591] 2022-01-20 11:43:40: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:40: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:40: jed-dev-01 compiled 7 got now: 4 speed: 24.2107 compile time 20175 produced code 6512679 client count: 10
[297591] 2022-01-20 11:43:40: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:40: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:40: HP-Z800 compiled 2 got now: 5 speed: 459.285 compile time 10072 produced code 2091388 client count: 5
[297591] 2022-01-20 11:43:40: penalizing local build for job 19
[297591] 2022-01-20 11:43:40: taking best installed HP-Z800 459.285
[297591] 2022-01-20 11:43:40: put 19 in joblist of HP-Z800
[297591] 2022-01-20 11:43:40: BEGIN: 19 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:41: END 14 status=0 in=0(0%) out=4319304(0%) real=3415 user=3210 sys=202 pfaults=115912 server=jed-dev-01
[297591] 2022-01-20 11:43:41: add_job_stats C++ 25 3210 10000 1199806 4319304 jed-dev-01 1345.58 329.805
[297591] 2022-01-20 11:43:41: NEW 20 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/results.cpp C++
[297591] 2022-01-20 11:43:41: pick_server 20 x86_64
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 compiled 8 got now: 3 speed: 26.7966 compile time 23385 produced code 7712485 client count: 10
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:41: HP-Z800 compiled 2 got now: 6 speed: 442.275 compile time 10072 produced code 2091388 client count: 6
[297591] 2022-01-20 11:43:41: penalizing local build for job 20
[297591] 2022-01-20 11:43:41: taking best installed HP-Z800 442.275
[297591] 2022-01-20 11:43:41: put 20 in joblist of HP-Z800
[297591] 2022-01-20 11:43:41: BEGIN: 20 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:41: END 16 status=0 in=0(0%) out=3783800(0%) real=3530 user=3306 sys=210 pfaults=111962 server=jed-dev-01
[297591] 2022-01-20 11:43:41: add_job_stats C++ 25 3306 10000 1051055 3783800 jed-dev-01 1144.53 328.333
[297591] 2022-01-20 11:43:41: END 1 status=0 in=4845195(20%) out=7883760(100%) real=7837 user=7214 sys=446 pfaults=128701 server=HP-Z800
[297591] 2022-01-20 11:43:41: add_job_stats C++ 25 7214 10000 2189933 7883760 HP-Z800 1092.84 743.027
[297591] 2022-01-20 11:43:41: END 13 status=0 in=0(0%) out=5803064(0%) real=4517 user=4278 sys=225 pfaults=138036 server=jed-dev-01
[297591] 2022-01-20 11:43:41: add_job_stats C++ 25 4278 10000 1611962 5803064 jed-dev-01 1356.49 335.029
[297591] 2022-01-20 11:43:41: NEW 21 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/schema.cpp C++
[297591] 2022-01-20 11:43:41: pick_server 21 x86_64
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 compiled 10 got now: 1 speed: 251.271 compile time 30969 produced code 10375502 client count: 8
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:41: HP-Z800 compiled 3 got now: 6 speed: 452.178 compile time 17286 produced code 4281321 client count: 7
[297591] 2022-01-20 11:43:41: slightly penalizing local build for job 21
[297591] 2022-01-20 11:43:41: taking best installed HP-Z800 452.178
[297591] 2022-01-20 11:43:41: put 21 in joblist of HP-Z800
[297591] 2022-01-20 11:43:41: NEW 22 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/set.cpp C++
[297591] 2022-01-20 11:43:41: pick_server 22 x86_64
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 compiled 10 got now: 1 speed: 31.4089 compile time 30969 produced code 10375502 client count: 9
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:41: HP-Z800 compiled 3 got now: 7 speed: 434.787 compile time 17286 produced code 4281321 client count: 7
[297591] 2022-01-20 11:43:41: penalizing local build for job 22
[297591] 2022-01-20 11:43:41: taking best installed HP-Z800 434.787
[297591] 2022-01-20 11:43:41: put 22 in joblist of HP-Z800
[297591] 2022-01-20 11:43:41: BEGIN: 21 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:41: BEGIN: 22 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:42: END 15 status=0 in=0(0%) out=4541592(0%) real=3818 user=3597 sys=215 pfaults=119785 server=jed-dev-01
[297591] 2022-01-20 11:43:42: add_job_stats C++ 26 3597 10000 1261553 4541592 jed-dev-01 1262.61 336.662
[297591] 2022-01-20 11:43:42: NEW 23 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/shared_realm.cpp C++
[297591] 2022-01-20 11:43:42: pick_server 23 x86_64
[297591] 2022-01-20 11:43:42: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:42: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:42: jed-dev-01 compiled 11 got now: 0 speed: 33.6662 compile time 34566 produced code 11637055 client count: 9
[297591] 2022-01-20 11:43:42: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:42: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:42: HP-Z800 compiled 3 got now: 8 speed: 417.395 compile time 17286 produced code 4281321 client count: 8
[297591] 2022-01-20 11:43:42: penalizing local build for job 23
[297591] 2022-01-20 11:43:42: taking best installed HP-Z800 417.395
[297591] 2022-01-20 11:43:42: put 23 in joblist of HP-Z800
[297591] 2022-01-20 11:43:42: BEGIN: 23 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:42: NEW 24 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] realm/object-store/thread_safe_reference.cpp C++
[297591] 2022-01-20 11:43:42: pick_server 24 x86_64
[297591] 2022-01-20 11:43:42: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:42: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:42: jed-dev-01 compiled 11 got now: 0 speed: 33.6662 compile time 34566 produced code 11637055 client count: 10
[297591] 2022-01-20 11:43:42: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:42: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:42: HP-Z800 compiled 3 got now: 9 speed: 400.004 compile time 17286 produced code 4281321 client count: 9
[297591] 2022-01-20 11:43:42: penalizing local build for job 24
[297591] 2022-01-20 11:43:42: taking best installed HP-Z800 400.004
[297591] 2022-01-20 11:43:42: put 24 in joblist of HP-Z800
[297591] 2022-01-20 11:43:42: BEGIN: 24 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:43: END 17 status=0 in=4734659(20%) out=2659528(100%) real=4771 user=4382 sys=291 pfaults=87576 server=HP-Z800
[297591] 2022-01-20 11:43:43: add_job_stats C++ 27 4382 10000 738757 2659528 HP-Z800 606.921 579.204
[297591] 2022-01-20 11:43:43: NEW 25 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] object-store/impl/collection_notifier.cpp C++
[297591] 2022-01-20 11:43:43: pick_server 25 x86_64
[297591] 2022-01-20 11:43:43: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:43: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:43: jed-dev-01 compiled 11 got now: 0 speed: 33.6662 compile time 34566 produced code 11637055 client count: 10
[297591] 2022-01-20 11:43:43: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:43: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:43: HP-Z800 compiled 4 got now: 9 speed: 311.811 compile time 21668 produced code 5020078 client count: 10
[297591] 2022-01-20 11:43:43: penalizing local build for job 25
[297591] 2022-01-20 11:43:43: taking best installed HP-Z800 311.811
[297591] 2022-01-20 11:43:43: put 25 in joblist of HP-Z800
[297591] 2022-01-20 11:43:43: BEGIN: 25 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:44: END 9 status=0 in=7081402(20%) out=6680968(100%) real=10124 user=9318 sys=545 pfaults=159322 server=HP-Z800
[297591] 2022-01-20 11:43:44: add_job_stats C++ 28 9318 10000 1855824 6680968 HP-Z800 716.996 443.807
[297591] 2022-01-20 11:43:44: END 18 status=0 in=4274322(20%) out=2332928(100%) real=3947 user=3610 sys=247 pfaults=74696 server=HP-Z800
[297591] 2022-01-20 11:43:44: add_job_stats C++ 28 3610 10000 648035 2332928 HP-Z800 646.24 326.22
[297591] 2022-01-20 11:43:44: NEW 26 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] object-store/impl/deep_change_checker.cpp C++
[297591] 2022-01-20 11:43:44: pick_server 26 x86_64
[297591] 2022-01-20 11:43:44: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:44: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:44: jed-dev-01 compiled 11 got now: 0 speed: 33.6662 compile time 34566 produced code 11637055 client count: 9
[297591] 2022-01-20 11:43:44: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:44: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:44: HP-Z800 compiled 6 got now: 8 speed: 183.254 compile time 34596 produced code 7523937 client count: 9
[297591] 2022-01-20 11:43:44: penalizing local build for job 26
[297591] 2022-01-20 11:43:44: taking best installed HP-Z800 183.254
[297591] 2022-01-20 11:43:44: put 26 in joblist of HP-Z800
[297591] 2022-01-20 11:43:44: BEGIN: 26 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:44: NEW 27 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] object-store/impl/list_notifier.cpp C++
[297591] 2022-01-20 11:43:44: pick_server 27 x86_64
[297591] 2022-01-20 11:43:44: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:44: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:44: jed-dev-01 compiled 11 got now: 0 speed: 33.6662 compile time 34566 produced code 11637055 client count: 10
[297591] 2022-01-20 11:43:44: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:44: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:44: HP-Z800 compiled 6 got now: 9 speed: 175.618 compile time 34596 produced code 7523937 client count: 9
[297591] 2022-01-20 11:43:44: penalizing local build for job 27
[297591] 2022-01-20 11:43:44: taking best installed HP-Z800 175.618
[297591] 2022-01-20 11:43:44: put 27 in joblist of HP-Z800
[297591] 2022-01-20 11:43:44: BEGIN: 27 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:45: END 21 status=0 in=3323728(19%) out=2125736(100%) real=3319 user=2984 sys=227 pfaults=56223 server=HP-Z800
[297591] 2022-01-20 11:43:45: add_job_stats C++ 29 2984 10000 590482 2125736 HP-Z800 712.378 215.924
[297591] 2022-01-20 11:43:45: NEW 28 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] object-store/impl/object_notifier.cpp C++
[297591] 2022-01-20 11:43:45: pick_server 28 x86_64
[297591] 2022-01-20 11:43:45: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:45: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:45: jed-dev-01 compiled 11 got now: 0 speed: 33.6662 compile time 34566 produced code 11637055 client count: 10
[297591] 2022-01-20 11:43:45: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:45: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:45: HP-Z800 compiled 7 got now: 9 speed: 116.241 compile time 37580 produced code 8114419 client count: 10
[297591] 2022-01-20 11:43:45: penalizing local build for job 28
[297591] 2022-01-20 11:43:45: taking best installed HP-Z800 116.241
[297591] 2022-01-20 11:43:45: put 28 in joblist of HP-Z800
[297591] 2022-01-20 11:43:45: BEGIN: 28 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:49: END 19 status=0 in=5295245(20%) out=3975488(100%) real=9273 user=8709 sys=445 pfaults=128944 server=HP-Z800
[297591] 2022-01-20 11:43:49: add_job_stats C++ 33 8709 10000 1567067 3975488 HP-Z800 456.48 209.153
[297591] 2022-01-20 11:43:49: END 27 status=0 in=4894840(20%) out=2984768(100%) real=5502 user=5026 sys=315 pfaults=97245 server=HP-Z800
[297591] 2022-01-20 11:43:49: add_job_stats C++ 33 5026 10000 876002 2984768 HP-Z800 593.865 205.739
[297591] 2022-01-20 11:43:50: NEW 29 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] object-store/impl/realm_coordinator.cpp C++
[297591] 2022-01-20 11:43:50: pick_server 29 x86_64
[297591] 2022-01-20 11:43:50: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:50: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:50: jed-dev-01 compiled 11 got now: 0 speed: 33.6662 compile time 34566 produced code 11637055 client count: 9
[297591] 2022-01-20 11:43:50: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:50: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:50: HP-Z800 compiled 9 got now: 8 speed: 85.1759 compile time 51315 produced code 10557488 client count: 9
[297591] 2022-01-20 11:43:50: penalizing local build for job 29
[297591] 2022-01-20 11:43:50: taking best installed HP-Z800 85.1759
[297591] 2022-01-20 11:43:50: put 29 in joblist of HP-Z800
[297591] 2022-01-20 11:43:50: BEGIN: 29 client=jed-dev-01(x86_64) server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:50: NEW 30 client=jed-dev-01 versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] object-store/impl/results_notifier.cpp C++
[297591] 2022-01-20 11:43:50: pick_server 30 x86_64
[297591] 2022-01-20 11:43:50: jed-dev-01 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:50: jed-dev-01 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:50: jed-dev-01 compiled 11 got now: 0 speed: 33.6662 compile time 34566 produced code 11637055 client count: 10
[297591] 2022-01-20 11:43:50: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:50: HP-Z800 is_eligible_now: 1 (jobs_okay 1, load_okay 1)
[297591] 2022-01-20 11:43:50: HP-Z800 compiled 9 got now: 9 speed: 81.6269 compile time 51315 produced code 10557488 client count: 9
[297591] 2022-01-20 11:43:50: penalizing local build for job 30
[297591] 2022-01-20 11:43:50: taking best installed HP-Z800 81.6269
[297591] 2022-01-20 11:43:50: put 30 in joblist of HP-Z800
HenryMiller1 commented 2 years ago

I don't think anyone left understands the algorithm. I too have noticed it has cases where it doesn't work well, but I haven't been annoyed enough to figure it out.

Sorry I cannot help, good luck.

On Thu, Jan 20, 2022, at 05:01, Jørgen Edelbo wrote:

A have a simple setup with two compiler servers, and after upgrading to v1.3 I find that jobs are primarily scheduled on one server.

It seems to be due to the face that when client_count is greater than maxJobs(), the client gets penalized, resulting in a situation where it will never get any jobs. In the trace file below you can see that jed-dev-01 never wins a job after job 17.

I am not sure where the problem lies. What does client_count represent? In this case it grows to 10,

[297591] 2022-01-20 11:43:16: ICECREAM scheduler 1.3.90 starting up, 
port 8765
[297591] 2022-01-20 11:43:16: scheduler ready
[297591] 2022-01-20 11:43:16: ignoring localhost lo for broadcast
[297591] 2022-01-20 11:43:16: broadcast wlp0s20f3 192.168.1.255
[297591] 2022-01-20 11:43:16: broadcast docker0 172.17.255.255
[297591] 2022-01-20 11:43:16: ignoring localhost lo for broadcast
[297591] 2022-01-20 11:43:16: broadcast wlp0s20f3 192.168.1.255
[297591] 2022-01-20 11:43:16: broadcast docker0 172.17.255.255
[297591] 2022-01-20 11:43:16: Received scheduler announcement from 
192.168.1.15:51849 (version 42, netname ICECREAM)
[297591] 2022-01-20 11:43:16: Received scheduler announcement from 
172.17.0.1:51849 (version 42, netname ICECREAM)
[297591] 2022-01-20 11:43:17: broadcast from 192.168.1.15:57618 
(version 42)
[297591] 2022-01-20 11:43:17: broadcast from 172.17.0.1:57618 (version 
42)
[297591] 2022-01-20 11:43:19: broadcast from 192.168.1.3:58655 (version 
42)
[297591] 2022-01-20 11:43:20: accepted 192.168.1.15
[297591] 2022-01-20 11:43:20: login jed-dev-01 protocol version: 42 
features: env_xz env_zstd []
[297591] 2022-01-20 11:43:23: accepted 192.168.1.3
[297591] 2022-01-20 11:43:23: login HP-Z800 protocol version: 42 
features: env_xz env_zstd [c161c43a4b8feaa9487be23f4e567534(x86_64), 
75b761cfe4904e62f1715ab5b1a1ba9a(x86_64), ]
[297591] 2022-01-20 11:43:33: NEW 1 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
sync/noinst/changeset_index.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 1 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly 
selected HP-Z800 load: 122 can install: x86_64
[297591] 2022-01-20 11:43:33: put 1 in joblist of HP-Z800
[297591] 2022-01-20 11:43:33: NEW 2 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
sync/noinst/compact_changesets.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 2 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly 
selected jed-dev-01 load: 449 can install: x86_64
[297591] 2022-01-20 11:43:33: put 2 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:33: BEGIN: 2 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:33: NEW 3 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
sync/noinst/client_history_impl.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 3 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly 
selected jed-dev-01 load: 449 can install: x86_64
[297591] 2022-01-20 11:43:33: put 3 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:33: BEGIN: 3 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:33: BEGIN: 1 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:33: NEW 4 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
sync/noinst/client_reset.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 4 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly 
selected HP-Z800 load: 122 can install: x86_64
[297591] 2022-01-20 11:43:33: put 4 in joblist of HP-Z800
[297591] 2022-01-20 11:43:33: BEGIN: 4 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:33: NEW 5 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
sync/noinst/protocol_codec.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 5 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly 
selected jed-dev-01 load: 449 can install: x86_64
[297591] 2022-01-20 11:43:33: put 5 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:33: BEGIN: 5 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:33: NEW 6 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
sync/noinst/client_reset_operation.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 6 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly 
selected jed-dev-01 load: 449 can install: x86_64
[297591] 2022-01-20 11:43:33: put 6 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:33: BEGIN: 6 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:33: NEW 7 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/sync/changeset_parser.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 7 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly 
selected jed-dev-01 load: 449 can install: x86_64
[297591] 2022-01-20 11:43:33: put 7 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:33: BEGIN: 7 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:33: NEW 8 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/sync/changeset_encoder.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 8 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly 
selected HP-Z800 load: 122 can install: x86_64
[297591] 2022-01-20 11:43:33: put 8 in joblist of HP-Z800
[297591] 2022-01-20 11:43:33: BEGIN: 8 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:33: NEW 9 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
sync/noinst/client_impl_base.cpp C++
[297591] 2022-01-20 11:43:33: pick_server 9 x86_64
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:33: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:33: no job stats - returning randomly 
selected HP-Z800 load: 122 can install: x86_64
[297591] 2022-01-20 11:43:33: put 9 in joblist of HP-Z800
[297591] 2022-01-20 11:43:33: BEGIN: 9 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:35: NEW 10 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/parser/driver.cpp C++
[297591] 2022-01-20 11:43:35: pick_server 10 x86_64
[297591] 2022-01-20 11:43:35: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:35: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:35: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:35: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:35: no job stats - returning randomly 
selected HP-Z800 load: 122 can install: x86_64
[297591] 2022-01-20 11:43:35: put 10 in joblist of HP-Z800
[297591] 2022-01-20 11:43:35: BEGIN: 10 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:36: END 6 status=0 in=0(0%) out=2594472(0%) 
real=2610 user=2435 sys=139 pfaults=91581 server=jed-dev-01
[297591] 2022-01-20 11:43:36: add_job_stats C++ 20 2435 10000 720686 
2594472 jed-dev-01 1065.49 1183.88
[297591] 2022-01-20 11:43:36: NEW 11 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/parser/keypath_mapping.cpp C++
[297591] 2022-01-20 11:43:36: pick_server 11 x86_64
[297591] 2022-01-20 11:43:36: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:36: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:36: jed-dev-01 compiled 1 got now: 4 speed: 
88.7909 compile time 2435 produced code 720686 client count: 10
[297591] 2022-01-20 11:43:36: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:36: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:36: HP-Z800 compiled 0 got now: 5 speed: 0 
compile time 0 produced code 0 client count: 5
[297591] 2022-01-20 11:43:36: penalizing local build for job 11
[297591] 2022-01-20 11:43:36: taking best installed jed-dev-01 88.7909
[297591] 2022-01-20 11:43:36: put 11 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:36: BEGIN: 11 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:36: END 2 status=0 in=0(0%) out=1904304(0%) 
real=3119 user=2916 sys=186 pfaults=88784 server=jed-dev-01
[297591] 2022-01-20 11:43:36: add_job_stats C++ 20 2916 10000 528973 
1904304 jed-dev-01 653.054 817.381
[297591] 2022-01-20 11:43:37: NEW 12 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
parser/generated/query_flex.cpp C++
[297591] 2022-01-20 11:43:37: pick_server 12 x86_64
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 compiled 2 got now: 4 speed: 
61.3036 compile time 5351 produced code 1249659 client count: 10
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:37: HP-Z800 compiled 0 got now: 5 speed: 0 
compile time 0 produced code 0 client count: 5
[297591] 2022-01-20 11:43:37: penalizing local build for job 12
[297591] 2022-01-20 11:43:37: taking best installed jed-dev-01 61.3036
[297591] 2022-01-20 11:43:37: put 12 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:37: BEGIN: 12 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:37: END 5 status=0 in=0(0%) out=2070328(0%) 
real=3325 user=3115 sys=155 pfaults=91109 server=jed-dev-01
[297591] 2022-01-20 11:43:37: add_job_stats C++ 21 3115 10000 575091 
2070328 jed-dev-01 664.632 646.616
[297591] 2022-01-20 11:43:37: NEW 13 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
parser/generated/query_bison.cpp C++
[297591] 2022-01-20 11:43:37: pick_server 13 x86_64
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 compiled 3 got now: 4 speed: 
48.4962 compile time 8466 produced code 1824750 client count: 10
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:37: HP-Z800 compiled 0 got now: 5 speed: 0 
compile time 0 produced code 0 client count: 5
[297591] 2022-01-20 11:43:37: penalizing local build for job 13
[297591] 2022-01-20 11:43:37: taking best installed jed-dev-01 48.4962
[297591] 2022-01-20 11:43:37: put 13 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:37: BEGIN: 13 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:37: END 7 status=0 in=0(0%) out=5887488(0%) 
real=3616 user=3422 sys=183 pfaults=107904 server=jed-dev-01
[297591] 2022-01-20 11:43:37: add_job_stats C++ 21 3422 10000 1635413 
5887488 jed-dev-01 1720.48 727.659
[297591] 2022-01-20 11:43:37: NEW 14 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/collection.cpp C++
[297591] 2022-01-20 11:43:37: pick_server 14 x86_64
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:37: jed-dev-01 compiled 4 got now: 4 speed: 
54.5744 compile time 11888 produced code 3460163 client count: 10
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:37: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:37: HP-Z800 compiled 0 got now: 5 speed: 0 
compile time 0 produced code 0 client count: 5
[297591] 2022-01-20 11:43:37: penalizing local build for job 14
[297591] 2022-01-20 11:43:37: taking best installed jed-dev-01 54.5744
[297591] 2022-01-20 11:43:37: put 14 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:37: BEGIN: 14 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:38: END 3 status=0 in=0(0%) out=5847544(0%) 
real=4263 user=4019 sys=192 pfaults=131870 server=jed-dev-01
[297591] 2022-01-20 11:43:38: add_job_stats C++ 22 4019 10000 1624317 
5847544 jed-dev-01 1454.97 639.276
[297591] 2022-01-20 11:43:38: END 11 status=0 in=0(0%) out=1854024(0%) 
real=1515 user=1378 sys=128 pfaults=57483 server=jed-dev-01
[297591] 2022-01-20 11:43:38: add_job_stats C++ 22 1378 10000 515006 
1854024 jed-dev-01 1345.45 485.926
[297591] 2022-01-20 11:43:38: NEW 15 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/dictionary.cpp C++
[297591] 2022-01-20 11:43:38: pick_server 15 x86_64
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 compiled 6 got now: 3 speed: 
39.4815 compile time 17285 produced code 5599486 client count: 9
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:38: HP-Z800 compiled 0 got now: 5 speed: 0 
compile time 0 produced code 0 client count: 5
[297591] 2022-01-20 11:43:38: penalizing local build for job 15
[297591] 2022-01-20 11:43:38: taking best installed jed-dev-01 39.4815
[297591] 2022-01-20 11:43:38: put 15 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:38: BEGIN: 15 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:38: NEW 16 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/list.cpp C++
[297591] 2022-01-20 11:43:38: pick_server 16 x86_64
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 compiled 6 got now: 4 speed: 
36.4444 compile time 17285 produced code 5599486 client count: 10
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:38: HP-Z800 compiled 0 got now: 5 speed: 0 
compile time 0 produced code 0 client count: 0
[297591] 2022-01-20 11:43:38: penalizing local build for job 16
[297591] 2022-01-20 11:43:38: taking best installed jed-dev-01 36.4444
[297591] 2022-01-20 11:43:38: put 16 in joblist of jed-dev-01
[297591] 2022-01-20 11:43:38: BEGIN: 16 client=jed-dev-01(x86_64) 
server=jed-dev-01(x86_64)
[297591] 2022-01-20 11:43:38: END 8 status=0 in=4645457(20%) 
out=3208768(100%) real=4625 user=4221 sys=231 pfaults=94615 
server=HP-Z800
[297591] 2022-01-20 11:43:38: add_job_stats C++ 22 4221 10000 891324 
3208768 HP-Z800 760.191 844.657
[297591] 2022-01-20 11:43:38: NEW 17 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/object.cpp C++
[297591] 2022-01-20 11:43:38: pick_server 17 x86_64
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:38: jed-dev-01 compiled 6 got now: 5 speed: 
33.4074 compile time 17285 produced code 5599486 client count: 10
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:38: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:38: HP-Z800 compiled 1 got now: 4 speed: 
553.567 compile time 4221 produced code 891324 client count: 5
[297591] 2022-01-20 11:43:38: penalizing local build for job 17
[297591] 2022-01-20 11:43:38: taking best installed HP-Z800 553.567
[297591] 2022-01-20 11:43:38: put 17 in joblist of HP-Z800
[297591] 2022-01-20 11:43:38: BEGIN: 17 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:40: END 12 status=0 in=0(0%) out=3287496(0%) 
real=3042 user=2890 sys=147 pfaults=102761 server=jed-dev-01
[297591] 2022-01-20 11:43:40: add_job_stats C++ 24 2890 10000 913193 
3287496 jed-dev-01 1137.54 322.809
[297591] 2022-01-20 11:43:40: END 4 status=0 in=5147606(20%) 
out=4320232(100%) real=6418 user=5851 sys=335 pfaults=132809 
server=HP-Z800
[297591] 2022-01-20 11:43:40: add_job_stats C++ 24 5851 10000 1200064 
4320232 HP-Z800 738.375 726.753
[297591] 2022-01-20 11:43:40: NEW 18 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/object_schema.cpp C++
[297591] 2022-01-20 11:43:40: pick_server 18 x86_64
[297591] 2022-01-20 11:43:40: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:40: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:40: jed-dev-01 compiled 7 got now: 4 speed: 
24.2107 compile time 20175 produced code 6512679 client count: 10
[297591] 2022-01-20 11:43:40: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:40: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:40: HP-Z800 compiled 2 got now: 4 speed: 
476.296 compile time 10072 produced code 2091388 client count: 5
[297591] 2022-01-20 11:43:40: penalizing local build for job 18
[297591] 2022-01-20 11:43:40: taking best installed HP-Z800 476.296
[297591] 2022-01-20 11:43:40: put 18 in joblist of HP-Z800
[297591] 2022-01-20 11:43:40: BEGIN: 18 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:40: NEW 19 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/object_store.cpp C++
[297591] 2022-01-20 11:43:40: pick_server 19 x86_64
[297591] 2022-01-20 11:43:40: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:40: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:40: jed-dev-01 compiled 7 got now: 4 speed: 
24.2107 compile time 20175 produced code 6512679 client count: 10
[297591] 2022-01-20 11:43:40: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:40: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:40: HP-Z800 compiled 2 got now: 5 speed: 
459.285 compile time 10072 produced code 2091388 client count: 5
[297591] 2022-01-20 11:43:40: penalizing local build for job 19
[297591] 2022-01-20 11:43:40: taking best installed HP-Z800 459.285
[297591] 2022-01-20 11:43:40: put 19 in joblist of HP-Z800
[297591] 2022-01-20 11:43:40: BEGIN: 19 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:41: END 14 status=0 in=0(0%) out=4319304(0%) 
real=3415 user=3210 sys=202 pfaults=115912 server=jed-dev-01
[297591] 2022-01-20 11:43:41: add_job_stats C++ 25 3210 10000 1199806 
4319304 jed-dev-01 1345.58 329.805
[297591] 2022-01-20 11:43:41: NEW 20 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/results.cpp C++
[297591] 2022-01-20 11:43:41: pick_server 20 x86_64
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 compiled 8 got now: 3 speed: 
26.7966 compile time 23385 produced code 7712485 client count: 10
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:41: HP-Z800 compiled 2 got now: 6 speed: 
442.275 compile time 10072 produced code 2091388 client count: 6
[297591] 2022-01-20 11:43:41: penalizing local build for job 20
[297591] 2022-01-20 11:43:41: taking best installed HP-Z800 442.275
[297591] 2022-01-20 11:43:41: put 20 in joblist of HP-Z800
[297591] 2022-01-20 11:43:41: BEGIN: 20 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:41: END 16 status=0 in=0(0%) out=3783800(0%) 
real=3530 user=3306 sys=210 pfaults=111962 server=jed-dev-01
[297591] 2022-01-20 11:43:41: add_job_stats C++ 25 3306 10000 1051055 
3783800 jed-dev-01 1144.53 328.333
[297591] 2022-01-20 11:43:41: END 1 status=0 in=4845195(20%) 
out=7883760(100%) real=7837 user=7214 sys=446 pfaults=128701 
server=HP-Z800
[297591] 2022-01-20 11:43:41: add_job_stats C++ 25 7214 10000 2189933 
7883760 HP-Z800 1092.84 743.027
[297591] 2022-01-20 11:43:41: END 13 status=0 in=0(0%) out=5803064(0%) 
real=4517 user=4278 sys=225 pfaults=138036 server=jed-dev-01
[297591] 2022-01-20 11:43:41: add_job_stats C++ 25 4278 10000 1611962 
5803064 jed-dev-01 1356.49 335.029
[297591] 2022-01-20 11:43:41: NEW 21 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/schema.cpp C++
[297591] 2022-01-20 11:43:41: pick_server 21 x86_64
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 compiled 10 got now: 1 speed: 
251.271 compile time 30969 produced code 10375502 client count: 8
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:41: HP-Z800 compiled 3 got now: 6 speed: 
452.178 compile time 17286 produced code 4281321 client count: 7
[297591] 2022-01-20 11:43:41: slightly penalizing local build for job 21
[297591] 2022-01-20 11:43:41: taking best installed HP-Z800 452.178
[297591] 2022-01-20 11:43:41: put 21 in joblist of HP-Z800
[297591] 2022-01-20 11:43:41: NEW 22 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/set.cpp C++
[297591] 2022-01-20 11:43:41: pick_server 22 x86_64
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:41: jed-dev-01 compiled 10 got now: 1 speed: 
31.4089 compile time 30969 produced code 10375502 client count: 9
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:41: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:41: HP-Z800 compiled 3 got now: 7 speed: 
434.787 compile time 17286 produced code 4281321 client count: 7
[297591] 2022-01-20 11:43:41: penalizing local build for job 22
[297591] 2022-01-20 11:43:41: taking best installed HP-Z800 434.787
[297591] 2022-01-20 11:43:41: put 22 in joblist of HP-Z800
[297591] 2022-01-20 11:43:41: BEGIN: 21 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:41: BEGIN: 22 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:42: END 15 status=0 in=0(0%) out=4541592(0%) 
real=3818 user=3597 sys=215 pfaults=119785 server=jed-dev-01
[297591] 2022-01-20 11:43:42: add_job_stats C++ 26 3597 10000 1261553 
4541592 jed-dev-01 1262.61 336.662
[297591] 2022-01-20 11:43:42: NEW 23 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/shared_realm.cpp C++
[297591] 2022-01-20 11:43:42: pick_server 23 x86_64
[297591] 2022-01-20 11:43:42: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:42: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:42: jed-dev-01 compiled 11 got now: 0 speed: 
33.6662 compile time 34566 produced code 11637055 client count: 9
[297591] 2022-01-20 11:43:42: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:42: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:42: HP-Z800 compiled 3 got now: 8 speed: 
417.395 compile time 17286 produced code 4281321 client count: 8
[297591] 2022-01-20 11:43:42: penalizing local build for job 23
[297591] 2022-01-20 11:43:42: taking best installed HP-Z800 417.395
[297591] 2022-01-20 11:43:42: put 23 in joblist of HP-Z800
[297591] 2022-01-20 11:43:42: BEGIN: 23 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:42: NEW 24 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
realm/object-store/thread_safe_reference.cpp C++
[297591] 2022-01-20 11:43:42: pick_server 24 x86_64
[297591] 2022-01-20 11:43:42: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:42: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:42: jed-dev-01 compiled 11 got now: 0 speed: 
33.6662 compile time 34566 produced code 11637055 client count: 10
[297591] 2022-01-20 11:43:42: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:42: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:42: HP-Z800 compiled 3 got now: 9 speed: 
400.004 compile time 17286 produced code 4281321 client count: 9
[297591] 2022-01-20 11:43:42: penalizing local build for job 24
[297591] 2022-01-20 11:43:42: taking best installed HP-Z800 400.004
[297591] 2022-01-20 11:43:42: put 24 in joblist of HP-Z800
[297591] 2022-01-20 11:43:42: BEGIN: 24 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:43: END 17 status=0 in=4734659(20%) 
out=2659528(100%) real=4771 user=4382 sys=291 pfaults=87576 
server=HP-Z800
[297591] 2022-01-20 11:43:43: add_job_stats C++ 27 4382 10000 738757 
2659528 HP-Z800 606.921 579.204
[297591] 2022-01-20 11:43:43: NEW 25 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
object-store/impl/collection_notifier.cpp C++
[297591] 2022-01-20 11:43:43: pick_server 25 x86_64
[297591] 2022-01-20 11:43:43: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:43: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:43: jed-dev-01 compiled 11 got now: 0 speed: 
33.6662 compile time 34566 produced code 11637055 client count: 10
[297591] 2022-01-20 11:43:43: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:43: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:43: HP-Z800 compiled 4 got now: 9 speed: 
311.811 compile time 21668 produced code 5020078 client count: 10
[297591] 2022-01-20 11:43:43: penalizing local build for job 25
[297591] 2022-01-20 11:43:43: taking best installed HP-Z800 311.811
[297591] 2022-01-20 11:43:43: put 25 in joblist of HP-Z800
[297591] 2022-01-20 11:43:43: BEGIN: 25 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:44: END 9 status=0 in=7081402(20%) 
out=6680968(100%) real=10124 user=9318 sys=545 pfaults=159322 
server=HP-Z800
[297591] 2022-01-20 11:43:44: add_job_stats C++ 28 9318 10000 1855824 
6680968 HP-Z800 716.996 443.807
[297591] 2022-01-20 11:43:44: END 18 status=0 in=4274322(20%) 
out=2332928(100%) real=3947 user=3610 sys=247 pfaults=74696 
server=HP-Z800
[297591] 2022-01-20 11:43:44: add_job_stats C++ 28 3610 10000 648035 
2332928 HP-Z800 646.24 326.22
[297591] 2022-01-20 11:43:44: NEW 26 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
object-store/impl/deep_change_checker.cpp C++
[297591] 2022-01-20 11:43:44: pick_server 26 x86_64
[297591] 2022-01-20 11:43:44: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:44: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:44: jed-dev-01 compiled 11 got now: 0 speed: 
33.6662 compile time 34566 produced code 11637055 client count: 9
[297591] 2022-01-20 11:43:44: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:44: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:44: HP-Z800 compiled 6 got now: 8 speed: 
183.254 compile time 34596 produced code 7523937 client count: 9
[297591] 2022-01-20 11:43:44: penalizing local build for job 26
[297591] 2022-01-20 11:43:44: taking best installed HP-Z800 183.254
[297591] 2022-01-20 11:43:44: put 26 in joblist of HP-Z800
[297591] 2022-01-20 11:43:44: BEGIN: 26 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:44: NEW 27 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
object-store/impl/list_notifier.cpp C++
[297591] 2022-01-20 11:43:44: pick_server 27 x86_64
[297591] 2022-01-20 11:43:44: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:44: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:44: jed-dev-01 compiled 11 got now: 0 speed: 
33.6662 compile time 34566 produced code 11637055 client count: 10
[297591] 2022-01-20 11:43:44: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:44: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:44: HP-Z800 compiled 6 got now: 9 speed: 
175.618 compile time 34596 produced code 7523937 client count: 9
[297591] 2022-01-20 11:43:44: penalizing local build for job 27
[297591] 2022-01-20 11:43:44: taking best installed HP-Z800 175.618
[297591] 2022-01-20 11:43:44: put 27 in joblist of HP-Z800
[297591] 2022-01-20 11:43:44: BEGIN: 27 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:45: END 21 status=0 in=3323728(19%) 
out=2125736(100%) real=3319 user=2984 sys=227 pfaults=56223 
server=HP-Z800
[297591] 2022-01-20 11:43:45: add_job_stats C++ 29 2984 10000 590482 
2125736 HP-Z800 712.378 215.924
[297591] 2022-01-20 11:43:45: NEW 28 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
object-store/impl/object_notifier.cpp C++
[297591] 2022-01-20 11:43:45: pick_server 28 x86_64
[297591] 2022-01-20 11:43:45: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:45: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:45: jed-dev-01 compiled 11 got now: 0 speed: 
33.6662 compile time 34566 produced code 11637055 client count: 10
[297591] 2022-01-20 11:43:45: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:45: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:45: HP-Z800 compiled 7 got now: 9 speed: 
116.241 compile time 37580 produced code 8114419 client count: 10
[297591] 2022-01-20 11:43:45: penalizing local build for job 28
[297591] 2022-01-20 11:43:45: taking best installed HP-Z800 116.241
[297591] 2022-01-20 11:43:45: put 28 in joblist of HP-Z800
[297591] 2022-01-20 11:43:45: BEGIN: 28 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:49: END 19 status=0 in=5295245(20%) 
out=3975488(100%) real=9273 user=8709 sys=445 pfaults=128944 
server=HP-Z800
[297591] 2022-01-20 11:43:49: add_job_stats C++ 33 8709 10000 1567067 
3975488 HP-Z800 456.48 209.153
[297591] 2022-01-20 11:43:49: END 27 status=0 in=4894840(20%) 
out=2984768(100%) real=5502 user=5026 sys=315 pfaults=97245 
server=HP-Z800
[297591] 2022-01-20 11:43:49: add_job_stats C++ 33 5026 10000 876002 
2984768 HP-Z800 593.865 205.739
[297591] 2022-01-20 11:43:50: NEW 29 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
object-store/impl/realm_coordinator.cpp C++
[297591] 2022-01-20 11:43:50: pick_server 29 x86_64
[297591] 2022-01-20 11:43:50: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:50: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:50: jed-dev-01 compiled 11 got now: 0 speed: 
33.6662 compile time 34566 produced code 11637055 client count: 9
[297591] 2022-01-20 11:43:50: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:50: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:50: HP-Z800 compiled 9 got now: 8 speed: 
85.1759 compile time 51315 produced code 10557488 client count: 9
[297591] 2022-01-20 11:43:50: penalizing local build for job 29
[297591] 2022-01-20 11:43:50: taking best installed HP-Z800 85.1759
[297591] 2022-01-20 11:43:50: put 29 in joblist of HP-Z800
[297591] 2022-01-20 11:43:50: BEGIN: 29 client=jed-dev-01(x86_64) 
server=HP-Z800(x86_64)
[297591] 2022-01-20 11:43:50: NEW 30 client=jed-dev-01 
versions=[75b761cfe4904e62f1715ab5b1a1ba9a(x86_64)] 
object-store/impl/results_notifier.cpp C++
[297591] 2022-01-20 11:43:50: pick_server 30 x86_64
[297591] 2022-01-20 11:43:50: jed-dev-01 is_eligible_ever: 1 (jobs_okay 
1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:50: jed-dev-01 is_eligible_now: 1 (jobs_okay 
1, load_okay 1)
[297591] 2022-01-20 11:43:50: jed-dev-01 compiled 11 got now: 0 speed: 
33.6662 compile time 34566 produced code 11637055 client count: 10
[297591] 2022-01-20 11:43:50: HP-Z800 is_eligible_ever: 1 (jobs_okay 1, 
version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, 
can_install 1, check_remote 1)
[297591] 2022-01-20 11:43:50: HP-Z800 is_eligible_now: 1 (jobs_okay 1, 
load_okay 1)
[297591] 2022-01-20 11:43:50: HP-Z800 compiled 9 got now: 9 speed: 
81.6269 compile time 51315 produced code 10557488 client count: 9
[297591] 2022-01-20 11:43:50: penalizing local build for job 30
[297591] 2022-01-20 11:43:50: taking best installed HP-Z800 81.6269
[297591] 2022-01-20 11:43:50: put 30 in joblist of HP-Z800

Btw. the scheduler cannot compile if you set DEBUG_SCHEDULER higher than 1.

-- Reply to this email directly or view it on GitHub: https://github.com/icecc/icecream/issues/593 You are receiving this because you are subscribed to this thread.

Message ID: @.***>

RedBeard0531 commented 2 years ago

Related #239

RedBeard0531 commented 2 years ago

Amusingly, Jørgen and I now work at the same office, although he filed this ticket independently. At one point I think I understood the algorithm and even patched it locally (in my last office) to prefer filling half a node's cores on the assumption that most CPUs are HyperThreading. I'll see if I can undust that corner of my brain and fix the issue he is seeing.

coolo commented 2 years ago

I don't think there will be a perfect algorithm. What was good for the old computers we used when creating icecream will be by no means good now :)

jnettlet commented 2 years ago

Started experimenting with icecream over the weekend for building on a cluster. Straight compiles like the linux kernel generally seem to do fine, however testing with Yocto proved to be a very different story. Mainly because the main builder for yocto is doing much more than just compilation, running configure, linking, packaging etc. For Linux and BSD at least my initial thoughts were that the node should be reporting back getloadavg() to the scheduler and the simplest metric for choosing a node would be MAX_JOBS - getloadavg(). If it is negative then don't scheduler that server, otherwise fill the difference.

This algorithm could benefit mixed load workers beyond just using nice to prefer local processes, by also being more adaptive if the nodes are doing other work. This would also allow simpler configuration since you wouldn't need to tune iceccd depending on the workload of a single node in the cluster.

jedelbo commented 2 years ago

I close this issue as the problem I experienced was due to a configuration error on my side.