search

icecc / icecream

Distributed compiler with a central scheduler to share build load
GNU General Public License v2.0
1.58k stars 250 forks source link

Local daemon/ Freebsd jail/Poudriere Usage #517

Closed darkfiberiru closed 4 years ago

darkfiberiru commented 4 years ago

Hello I'm looking at using icecc four freebsd poudriere builds(freebsd ports tree 32000+ packages). Obviously building that much software takes a lot of resources.

This is a personal project but if I get it working there's a good chance it will be used at my employer iXsystems and other places.

One of the big complaints from poudriere mainteners when distcc was brought up in the past was that poudriere jails are a clean environment for reproducibility including no network connection unless default settings are overridden.

I saw that you can talk to a local daeamon via sockets. Will that then handle all traffic and proxy everything?

Ideal local icecc daemon on host that is talked to over a unix socket that is passed into jail would be great. Or a way to proxy out to host I guess is an alternative.

HenryMiller1 commented 4 years ago

I use icecc in a docker all the time. Just make sure the one socket is forwarded from the jail to the host. I think the default is some unix socket in /var, but it will fall back to an ip socket to localhost, but best is if you can figure out what socket that is. Iceccd runs on the host and communicates with the network. I thought I put the required setup in the readme,but I don't see it.

If you don't figure it out by Monday I'll look up what we do at work.

It would be nice if you can get tests running in freebsd and help us by setting up a ci build.

-- Henry Miller hank@millerfarm.com

On Sat, Feb 22, 2020, at 11:41 AM, darkfiberiru wrote:

Hello I'm looking at using icecc four freebsd poudriere builds(freebsd ports tree 32000+ packages). Obviously building that much software takes a lot of resources.

This is a personal project but if I get it working there's a good chance it will be used at my employer iXsystems and other places.

One of the big complaints from poudriere mainteners when distcc was brought up in the past was that poudriere jails are a clean environment for reproducibility including no network connection unless default settings are overridden.

I saw that you can talk to a local daeamon via sockets. Will that then handle all traffic and proxy everything?

Ideal local icecc daemon on host that is talked to over a unix socket that is passed into jail would be great. Or a way to proxy out to host I guess is an alternative.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/icecc/icecream/issues/517?email_source=notifications&email_token=ACHSQERMLQH2GTGRJDKVWS3REFPUJA5CNFSM4KZTG25KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IPP3N5Q, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHSQEUGE5OCRVPA5UEKH2TREFPUJANCNFSM4KZTG25A.

llunak commented 4 years ago

I saw that you can talk to a local daeamon via sockets. Will that then handle all traffic and proxy everything?

No. Communication related to remote builds themselves (sending the build job, receiving the result) is done directly icecc <-> remote iceccd. And if icecc is jailed, then even the icecc <-> local iceccd communication is done using a localhost socket, because icecc has no way to reach the unix socket. So you'll need to forward outgoing connections to the local iceccd and remote iceccds. I have no practical experience with that though.

darkfiberiru commented 4 years ago

Local host sockets can be passed through to FreeBSD jails with out any issues but sounds like I will need to figure out a solution for the icecc to remote-iceccd portion that was my biggest question with this.

llunak commented 4 years ago

Technically it would be possible to send that data via the daemon, but normally that's a large amount of data, so doing that would presumably affect performance. So I really suggest just forwarding the network traffic, and I don't see a big difference between talking to a local or remote iceccd. I'm going to close this, unless you have really strong reasons.