icecc / icecream

Distributed compiler with a central scheduler to share build load
GNU General Public License v2.0
1.6k stars 252 forks source link

Would like icecc to report the farm job limit #516

Open thiagomacieira opened 4 years ago

thiagomacieira commented 4 years ago

I am using Icecream on a laptop and half the time I am not connected to the network where there's a farm. I have to keep MAKEFLAGS set to a low count so I won't hose down my system if I accidentally launch 130 compile jobs. SImilarly, I have a hardcoded value of the job limit for the farm.

Can we get an icecc option that will print the cluster's job limit? That way, I can do something like:

function make() {
    command make -j`icecc --job-limit` "$@"
}
HenryMiller1 commented 4 years ago

You can get that from the API, but it isn't as useful as you want. If nodes come online or go away while building end up in the same situation, a job count mismatched to network. Same thing happens if someone else starts a build job while your build is running.

For this to be useful you really need to teach make to ask the network regularly and adjust in real time. I'm not sure how to get such patches approved (if I was maintainer I'd reject them because for non icecc users they are no help and the code complexity isn't worth it. If you wish to try don't forget you need a way to figure out which build gets priority,and how to pass things off when nothing distributable is coming up. Easy to say,but not trivial to figure out.

-- Henry Miller hank@millerfarm.com

On Fri, Feb 14, 2020, at 2:12 PM, Thiago Macieira wrote:

I am using Icecream on a laptop and half the time I am not connected to the network where there's a farm. I have to keep MAKEFLAGS set to a low count so I won't hose down my system if I accidentally launch 130 compile jobs. SImilarly, I have a hardcoded value of the job limit for the farm.

Can we get an icecc option that will print the cluster's job limit? That way, I can do something like:

function make() { command make -jicecc --job-limit "$@" }

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/icecc/icecream/issues/516?email_source=notifications&email_token=ACHSQEWBZDCA6ACQ7I6YASDRC33KJA5CNFSM4KVPAB52YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INVLXZQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHSQEQN7BHD2UYUIZ6TU5DRC33KJANCNFSM4KVPAB5Q.

thiagomacieira commented 4 years ago

The monitor does show the total, so I don't see how the information change while other jobs are happening.

Anyway, a better solution would be to have an icemake, which offers a GNU make-compatible job server to the child makes. It can control how many local jobs are running and how many are remote (not perfect: it wouldn't know whether the next job is started with icecc or not, so it could leave network capacity unused if there are too many local jobs running).

But that wouldn't work for other buildsystems like ninja, that don't have a job server. I still want to know the job limit in my icecream network.

llunak commented 4 years ago

I still want to know the job limit in my icecream network.

Why? I know it looks like you do, but I think you realistically don't. Icecream doesn't allow more compile jobs than the system can handle, so you don't need it for compile jobs (and in case you do, that's a bug that should be fixed). And for non-compile jobs, Icecream shouldn't matter, they can overload your system even with Icecream running[*], so again you shouldn't need it. So why exactly do you think you need it?

[*] There's icemon to help with that case.

See also #449 or https://github.com/icecc/icecream/blob/master/tests/test.sh#L1893 for how to do it if you insist you need to know. I'm just not convinced this needs to be officially supported.

thiagomacieira commented 4 years ago

Yes, it won't protect against non-icecream jobs, but I can modify some Makefiles to have prepend "icerun" to mitigate the issue. But the point is that if I am not in the office network, there's no need to put pressure on my system with -j150.

llunak commented 4 years ago

What pressure exactly? I've just run -j1000 build of LibreOffice and I could barely notice (which means I did 'ps ux | grep icecc | wc -l' and then compared 'free' output, which showed about 2GiB difference). If your system can handle the build with icecream enabled, why shouldn't handle it with icecream disabled? I still fail to see an actual practical problem.

milianw commented 4 years ago

@thiagomacieira this is how I'm handling this myself: https://github.com/milianw/shell-helpers/blob/master/icecream_or_no_icecream.sh -> also add symlinks for gcc, g++, clang, clang++ to that script https://github.com/milianw/shell-helpers/blob/master/icecream_is_available.sh https://github.com/milianw/shell-helpers/blob/master/make https://github.com/milianw/shell-helpers/blob/master/ninja

fgouget commented 4 years ago

icecream_is_available.sh is interesting. It mishandles client that are being run with --no-remote (which would be ok if this does not happen in your setup). When given --no-remote the client seems to be reported as using port 0. So a fix would be something like this (I also reduced the number of forked processes):

expr $((echo "listcs"; sleep $timeout_1; echo "quit") | timeout $timeout_2 telnet $ICECREAM_SCHEDULER_HOST 8766 2> /dev/null | sed -E 's~^.:10245) . jobs=[0-9]+/([0-9]+) .*$~\1 + ~' -e t -e d) 0

This assumes one uses the default 10245 port. Alternatively one could use a grep -v ':0)'