search

icecc / icecream

Distributed compiler with a central scheduler to share build load
GNU General Public License v2.0
1.6k stars 252 forks source link

icecc-scheduler - Option to only link files #481

Closed dioguerra closed 4 years ago

dioguerra commented 5 years ago

Add an option to icecc-scheduler to only use compute to link the compiled files.

This makes sence for a distributed containerized environment where i want to reserve the scheduler container to link all the files and spawn N container workers to compile the code.

dioguerra commented 5 years ago

/kind feature

llunak commented 5 years ago

Sorry, I don't understand what you mean (or how is this different from #482 ).

dioguerra commented 5 years ago

The idea here is that, i can use the scheduler only to run linking.

As it works now, i launch the daemon on the scheduler and it assembles stuff, until i have enough worker nodes doing the assembly to overload the master with link jobs.

This allows me to reduce the scheduler compute resources to a minimum of (lets say, 1 core) and then scale vertically.

This is not that usefull though if you have https://github.com/icecc/icecream/issues/482

You can close if you want.

llunak commented 5 years ago

I think you're confusing scheduler with the client maching submitting jobs. A scheduler can be any machine that runs the extra daemon coordinating build jobs. Either way, the scheduler tries to prefer remote nodes in case the compilation is parallel enough, in order to allow the submitter to spend the resources on doing jobs that cannot be distributed, such as linking. So in general this should already work. If you really want to limit usage on the submitter, you can configure its daemon to not allow remote jobs or to limit its allowed number of jobs.

dioguerra commented 5 years ago

As far as i undestand, only one scheduler per network might be used at one time (its a 1 to many scheduler workers architecture)

Can i compile multiple projects at the same time, using the same scheduler? I tried it, didn't work. Can i use 2 scheduler nodes to coordinate the workers together? How would this work if i don't have Broadcast available?

HenryMiller1 commented 5 years ago

1 scheduler, many clients. Any client can compile at any time, even if others already are. This is a common use case and generally works well. You would need to give more details on why it didn't work for you, it works for me.

-- Henry Miller hank@millerfarm.com

On Wed, Jul 17, 2019, at 8:26 AM, Diogo Guerra wrote:

As far as i undestand, only one scheduler per network might be used at one time (its a 1 to many scheduler workers architecture)

Can i compile multiple projects at the same time, using the same scheduler? I tried it, didn't work. Can i use 2 scheduler nodes to coordinate the workers together? How would this work if i don't have Broadcast available?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/icecc/icecream/issues/481?email_source=notifications&email_token=ACHSQETECG7H77I2BTKGXULP74MWPA5CNFSM4IBHYII2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2EF2UA#issuecomment-512253264, or mute the thread https://github.com/notifications/unsubscribe-auth/ACHSQEUQJLXWPIKZYTANW4DP74MWPANCNFSM4IBHYIIQ.