search

icecc / icecream

Distributed compiler with a central scheduler to share build load
GNU General Public License v2.0
1.6k stars 252 forks source link

Icecream not distributing tasks to remote nodes #582

Open jc-lab opened 3 years ago

jc-lab commented 3 years ago

icecc version : 1.3.1

Probably the same as issue #198. Build with ICECC_TEST_REMOTEBUILD=1, but only build on local machine.

local log:

[6] 2021-05-12 05:06:47: accepted 9 127.0.0.1 as 11222
[6] 2021-05-12 05:06:47: get_native_env for /usr/bin/gcc (/usr/bin/gcc,/usr/bin/g++)
[6] 2021-05-12 05:06:47: get_native_env /var/tmp/icecc-envs/native/3e7a623641870f94f89309ed82d0aec7.tar.gz (:/usr/bin/gcc)
[6] 2021-05-12 05:06:47: send JobLocalBeginMsg to client
[6] 2021-05-12 05:06:47: pushed local job 11222
[6] 2021-05-12 05:06:47: accepted 10 127.0.0.1 as 11223
[6] 2021-05-12 05:06:47: get_native_env for /usr/bin/g++ (/usr/bin/gcc,/usr/bin/g++)
[6] 2021-05-12 05:06:47: get_native_env /var/tmp/icecc-envs/native/3e7a623641870f94f89309ed82d0aec7.tar.gz (:/usr/bin/gcc)
[6] 2021-05-12 05:06:47: accepted 11 127.0.0.1 as 11224
[6] 2021-05-12 05:06:47: get_native_env for /usr/bin/g++ (/usr/bin/gcc,/usr/bin/g++)
[6] 2021-05-12 05:06:47: get_native_env /var/tmp/icecc-envs/native/3e7a623641870f94f89309ed82d0aec7.tar.gz (:/usr/bin/gcc)
[6] 2021-05-12 05:06:47: send JobLocalBeginMsg to client
[6] 2021-05-12 05:06:47: pushed local job 11223
[6] 2021-05-12 05:06:47: send JobLocalBeginMsg to client
[6] 2021-05-12 05:06:47: pushed local job 11224
[6] 2021-05-12 05:06:47: accepted 12 127.0.0.1 as 11225
[6] 2021-05-12 05:06:47: get_native_env for /usr/bin/g++ (/usr/bin/gcc,/usr/bin/g++)
[6] 2021-05-12 05:06:47: get_native_env /var/tmp/icecc-envs/native/3e7a623641870f94f89309ed82d0aec7.tar.gz (:/usr/bin/gcc)
[6] 2021-05-12 05:06:47: send JobLocalBeginMsg to client
[6] 2021-05-12 05:06:47: pushed local job 11225
[6] 2021-05-12 05:06:47: accepted 13 127.0.0.1 as 11226
[6] 2021-05-12 05:06:47: get_native_env for /usr/bin/g++ (/usr/bin/gcc,/usr/bin/g++)
[6] 2021-05-12 05:06:47: get_native_env /var/tmp/icecc-envs/native/3e7a623641870f94f89309ed82d0aec7.tar.gz (:/usr/bin/gcc)
[6] 2021-05-12 05:06:47: send JobLocalBeginMsg to client
[6] 2021-05-12 05:06:47: pushed local job 11226
[6] 2021-05-12 05:06:47: accepted 14 127.0.0.1 as 11227
[6] 2021-05-12 05:06:47: get_native_env for /usr/bin/gcc (/usr/bin/gcc,/usr/bin/g++)
[6] 2021-05-12 05:06:47: get_native_env /var/tmp/icecc-envs/native/3e7a623641870f94f89309ed82d0aec7.tar.gz (:/usr/bin/gcc)
[6] 2021-05-12 05:06:47: send JobLocalBeginMsg to client
[6] 2021-05-12 05:06:47: pushed local job 11227
[6] 2021-05-12 05:06:47: accepted 15 127.0.0.1 as 11228
[6] 2021-05-12 05:06:47: get_native_env for /usr/bin/g++ (/usr/bin/gcc,/usr/bin/g++)
[6] 2021-05-12 05:06:47: get_native_env /var/tmp/icecc-envs/native/3e7a623641870f94f89309ed82d0aec7.tar.gz (:/usr/bin/gcc)
[6] 2021-05-12 05:06:47: accepted 16 127.0.0.1 as 11229
[6] 2021-05-12 05:06:47: get_native_env for /usr/bin/g++ (/usr/bin/gcc,/usr/bin/g++)
[6] 2021-05-12 05:06:47: get_native_env /var/tmp/icecc-envs/native/3e7a623641870f94f89309ed82d0aec7.tar.gz (:/usr/bin/gcc)
[6] 2021-05-12 05:06:47: handle_end 11226 127.0.0.1

scheduler log:

[1] 2021-05-12 08:52:13: machine-1 is busy installing since 20 seconds.
[1] 2021-05-12 08:52:13: machine-1 is_eligible_now: 0 (jobs_okay 1, load_okay 1)
[1] 2021-05-12 08:52:13: machine-1 not eligible
[1] 2021-05-12 08:52:13: overloaded developer-pc 7/6 jobs, load:342
[1] 2021-05-12 08:52:13: machine-1 is_eligible_ever: 1 (jobs_okay 1, version_okay 1, features_okay 1, chroot_or_local 1, accepting 1, can_install [1] 2021-05-12 08:52:13: machine-1 is busy installing since 20 seconds.
0, check_remote 1)
[1] 2021-05-12 08:52:13: No suitable host found, delaying

sundae:

Scheduler: ...Netname: ICECREAM
Servers: Total:2 Available:2 Active:0
Total: Remote:0 Local:11230
Jobs: Maximum:14 Active:0 Local:0 Pending:0
      [              ]

↓ ID NAME                     IN    CUR MAX JOBS       OUT   LOCAL ACTIVE PENDING SPEED
+ 8  machine-1                0     0   8   [        ] 0     0     0      0       0
+ 9  developer-pc             0     0   6   [      ]   0     11230 0      0       0
jc-lab commented 3 years ago

Due to network environment, the IPs of two different nodes appear to be the same IP from the perspective of the scheduler. Could this be the problem? Temporarily tested it on the same network and it works fine. If this is the cause, Is there any way to fix the advertising IP in such a network environment?

jc-lab commented 3 years ago

[6254] 2021-05-15 20:59:13: f4b723fcacc6 : host_platform : [6254] 2021-05-15 20:59:13: put 420 in joblist of f4b723fcacc6 (will install now) [6254] 2021-05-15 20:59:13: f4b723fcacc6 : gotit : 0 (This is what I added) [6254] 2021-05-15 20:59:13: setBusyInstalling: f4b723fcacc6 : 1621079953 (This is what I added) [6254] 2021-05-15 20:59:15: END 420 status=118 [6254] 2021-05-15 20:59:15: handle_local_job /work/Bootstrap.cmk/cmDepends.o 621 [6254] 2021-05-15 20:59:16: handle_local_job_done 621 [6254] 2021-05-15 20:59:16: NEW 422 client=6403de00d158 versions=[3e7a623641870f94f89309ed82d0aec7(x86_64)] work/Source/cmDependsC.cxx C++ 0 [6254] 2021-05-15 20:59:16: pick_server 422 x86_64

srogatch commented 3 years ago

For me icecc is not running anything remotely either. The debug output on the client is:

ICECC[2024020] 2021-08-11 00:14:16: connect(Error: Connection refused)
ICECC[2024020] 2021-08-11 00:14:16: connect failed on /var/run/icecc/iceccd.socket
ICECC[2024020] 2021-08-11 00:14:16: connect(Error: No such file or directory)
ICECC[2024020] 2021-08-11 00:14:16: connect failed on /var/run/iceccd.socket
ICECC[2024020] 2021-08-11 00:14:16: connect(Error: Connection refused)
ICECC[2024020] 2021-08-11 00:14:16: connect failed on /home/serge/.iceccd.socket
ICECC[2024020] 2021-08-11 00:14:16: connect(Error: Connection refused)
ICECC[2024020] 2021-08-11 00:14:16: connect failed on 127.0.0.1
ICECC[2024020] 2021-08-11 00:14:16: no local daemon found

The logs are mostly empty.

themightyoarfish commented 1 year ago

Same problem here, anyone know what to do?

ICECC[1027145] 2023-02-08 15:36:45: connect(Error: Connection refused)
ICECC[1027145] 2023-02-08 15:36:45: connect failed on /var/run/icecc/iceccd.socket
ICECC[1027145] 2023-02-08 15:36:45: connect(Error: No such file or directory)
ICECC[1027145] 2023-02-08 15:36:45: connect failed on /var/run/iceccd.socket
ICECC[1027145] 2023-02-08 15:36:45: connect(Error: Connection refused)
ICECC[1027145] 2023-02-08 15:36:45: connect failed on /home/rasmus/.iceccd.socket
ICECC[1027145] 2023-02-08 15:36:45: connect(Error: Connection refused)
ICECC[1027145] 2023-02-08 15:36:45: connect failed on 127.0.0.1
ICECC[1027145] 2023-02-08 15:36:45: -{march,mpcu,mtune}=native optimizes for local machine, building locally
ICECC[1027145] 2023-02-08 15:36:45: no local daemon found

this is on

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:        20.04
Codename:       focal
themightyoarfish commented 1 year ago

I do not have these issues with icecc built from source by the way. Likely the apt package is very old.