search

mozilla / sccache

Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible. Sccache has the capability to utilize caching in remote storage environments, including various cloud storage options, or alternatively, in local storage.
Apache License 2.0
5.85k stars 552 forks source link

Setting up Distributed sccache on EC2 instances. #974

Open rahulbansal16 opened 3 years ago

rahulbansal16 commented 3 years ago

Hi,

I am trying to set up the sccache on the Ec2 instances and I am not able to get it up and running. I have tried few techniques to make it work but I have not got any success.

I thought of documenting all the steps that I took and will keep on posting the new ones.

Intended Setup I want to put the scheduler server on a dedicated machine and connect the build servers to the scheduler. I want the scheduler to be accessible by the General Public with the required token.

This means that the build server should be running on the public IP instead of the private one.

Approaches tried

  1. Setting the public_addr value to the public IP in the build server

    The public IP needs to be associated with the OS for this approach to work. I did not have much idea about doing that so I thought of trying the next approach. Refer to the StackOverflow question for more details.

  2. Listening on the Private IP

    I tried listening on the Private IP of the build server to at least get the sccache working for the subnet. I set the value of the public_addr value to 172.23.23.22:10501. This leads to an error of invalid_bearer_token_mismatched_address. I have set the scheduler auth type to DANGEROUSLY_INSECURE.

  3. Setting the public_addr to 127.0.0.1:10501 and doing a Port Forwarding from Scheduler and Client

    In this, I thought of keeping the public_addr value to 127.0.0.1:10501 and doing a port forwarding from the client and scheduler to build server port 10501. This gives the error channel 3: open failed: connect failed: Connection refused

  4. Hardcoding the ip addresses

    I hardcoded the address in the line https://github.com/mozilla/sccache/blob/915e977bc3d7339030119462db3b04b856cf08ee/src/dist/http.rs#L772 with the public_addr "0.0.0.0:10501" to make it possible to expose the port outside VM and returning server_id in the line https://github.com/mozilla/sccache/blob/915e977bc3d7339030119462db3b04b856cf08ee/src/dist/http.rs#L719

    EDIT I got it working via the 4th approach

luser commented 3 years ago

I want to put the scheduler server on a dedicated machine and connect the build servers to the scheduler. I want the scheduler to be accessible by the General Public with the required token.

FYI, sccache's distributed compilation support was not designed to be used with untrusted users. While it does sandbox compile jobs on the build servers, I'd caution against relying on that to provide access to arbitrary users.

rahulbansal16 commented 3 years ago

@luser Thanks for the information. I am trying to setup a one-click distributed sccache setup for the organizations. Just for testing purposes, I was trying to provide access to the untrusted users.

aidanhs commented 3 years ago

This is essentially a feature request to allow build servers to inform the scheduler of an address that they can be connected to at, rather than (re)using the listen address. This seems reasonable.