Remove hard dep on Sourcegraph's internal dev-private repo

[ghost]

Question description

Hello, first of all I wanted to thank you for such an amazing project

I'm trying to deploy a self-hosted instance of sourcegraph, just to try and see how it goes for us, and if it fits our needs.

The question that I have is regarding the Server image, the one that you find here: https://hub.docker.com/r/sourcegraph/server

The thing is, as far as I see on the repository the server image is built from the enterprise section, which at the same time requires to clone a private repo with build dependencies which we, normal users, don't have access to.

The reason I'm doing this is to set some security configurations by default at built time:

  "auth.public": false,
  "auth.sessionExpiry": "12h",
  "auth.providers": [
    {
      "type": "builtin",
      "allowSignup": false
    }
  ]

And to somehow set an admin account instead of allowing the first user to become admin automatically, which, in case of a server restart -we don't use persistent storage-, could allow any user to become admin and disclose repositories information

We firmly believe in open source, so revealing our repositories to the public domain is not a problem -many of our repos are open source-. However the repositories that we'll be hosting are not ours.

Additional context

I know you can set those options on the management console. But it's not an option for us as a company since we follow a philosophy of inmutable and stateless infrastructure, so the Docker image needs to be deployed to servers completely configured, with no hands involved after deployment.

thank you so much for your time and help in advance

[sqs]

We definitely intend to and want to support this use case (building the server images yourself).

requires to clone a private repo with build dependencies which we, normal users, don't have access to

I believe this is referring to our dev-private repository, which is used in https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/enterprise/dev/start.sh. This doesn't actually contain secret code/deps that are necessary for building Sourcegraph (there aren't any), it just contains our internal dogfooding configuration (such as the token for our dev GitLab/GitHub/Phabricator instances).

You can recreate that by creating a sibling dir to sourcegraph called dev-private with the following contents:

enterprise/dev/env:

# Nothing needed here. For us, this sets various API keys.

enterprise/dev/external-services-config.json (the object keys are external service kind strings, GITHUB/GITLAB/etc. are also valid):

{
  "PHABRICATOR": [
    {
      "url": "https://phabricator.example.org",
      ...

enterprise/dev/critical-config.json:

// The Sourcegraph critical configuration for development servers.
{
  "externalURL": "http://localhost:3080",
  ...

enterprise/dev/site-config.json:

{
  // Site config goes here
  ...
}

Then you should be able to run enterprise/dev/start.sh. Let me know if you have any issues (I just typed this up and haven't completely verified it).

---

Leaving this issue open because we SHOULD remove the hard dependency on this dev-private repo and make this process actually supported and documented (not requiring these manual steps).

[slimsag]

The advice by @sqs above is good of you are hard-set on building the sourcegraph/server image. However, I would suggest that there is perhaps a better way to achieve what you are trying to achieve.

1. Start a sourcegraph/server using the docker run command on docs.sourcegraph.com.
2. Create the admin account as you desire
3. Make your configuration changes through the web UI and management console.
4. Stop that sourcegraph/server instance.
5. Create your own Dockerfile with FROM sourcegraph/server and have it bundle in the Sourcegraph data from your previously ran instance, e.g.:

FROM sourcegraph/server
COPY ~/.sourcegraph/config /etc/sourcegraph
COPY ~/.sourcegraph/data /var/opt/sourcegraph

Now any time you run your own docker image produced by the above Dockerfile, your server will start out with the admin account you created and the configuration you set up (obviously, you would also want to avoid publishing that image to docker hub as it would contain the admin credentials inside of it).

[ghost]

Thank you so much guys, both methods worked perfectly, the first allowed me to compile and run both the DEV enviroment as well as to build the image myself, (I had to set some enviroment variables first though)

The second method is pretty good too and I paste it here for it could be useful to someone else:

set -ex

# delete the persistent volumes
sudo rm -rf ~/.sourcegraph/config
sudo rm -rf ~/.sourcegraph/data

# run an instance
sudo docker run \
        --publish 7080:7080 \
        --publish 2633:2633 \
        --volume ~/.sourcegraph/config:/etc/sourcegraph \
        --volume ~/.sourcegraph/data:/var/opt/sourcegraph \
        --rm \
    sourcegraph/server:3.1.1

# do your configs and grab the admin credentials

After this, lets copy the persistent volumes, change some permissions and finally constructs a nice directory structure relative to /

# stop server first so everything is dumped down to persistent volumes

set -ex

rm -rf etc
rm -rf var

sudo cp -a ~/.sourcegraph .
sudo chown --recursive <<<<Your user here>>>> .sourcegraph

mkdir etc
mv .sourcegraph/config etc/
mv etc/config etc/sourcegraph

mkdir var
mkdir var/opt
mv .sourcegraph/data var/opt/
mv var/opt/data var/opt/sourcegraph
rm -rf var/opt/sourcegraph/repos
rm -rf var/opt/sourcegraph/zoekt

rm -rf sourcegraph_overrides.zip
zip -r sourcegraph_overrides.zip etc/ var/

rm -rf .sourcegraph
rm -rf etc/
rm -rf var/

we have now a sourcegraph_overrides.zip which contains the changes made. we can build our image from the original, we have to give some permissions to postgres

FROM sourcegraph/server:3.1.1

COPY sourcegraph_overrides.zip /

RUN cd / \
    && unzip /sourcegraph_overrides.zip \
    && chown --recursive postgres:postgres /var/opt/sourcegraph/postgresql \
    && chown postgres:root /var/opt/sourcegraph/postgresql \
    && chmod -R 777 /etc/sourcegraph \
    && chmod -R 777 /var/opt/sourcegraph \
    && chmod -R 0750 /var/opt/sourcegraph/postgresql

finally build and run stateless:

# build and run
if sudo docker build --tag "test:latest" .
then
    sudo docker run \
            --publish 7080:7080 \
            --publish 2633:2633 \
            --rm \
        test:latest
fi

Just playing around with this things, do we currently have a brute force protection? like incremental waiting times on fail, or maximum number of failed attempts until getting banned from server?

Are we safe from SQL injections, XSS, MITM and other from the infinite CWE list ?

[keegancsmith]

Is this issue resolved?

[slimsag]

No

On Mon, Jan 13, 2020, 3:55 AM Keegan Carruthers-Smith < notifications@github.com> wrote:

Is this issue resolved?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sourcegraph/sourcegraph/issues/2417?email_source=notifications&email_token=AAYGWOA7T6LREJUS5JLSF3DQ5RCC3A5CNFSM4GZJJ2KKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIYIY3Q#issuecomment-573607022, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYGWOAZTI434DAGIRF74FLQ5RCC3ANCNFSM4GZJJ2KA .