Instructions for deploying personal borg server

[choudhary001]

Hi,

I want to deploy a personal borg server but I couldn't find any instructions in this repository. This section in readme https://github.com/ok-borg/borg#how-does-borg-work asks to see daemon folder but couldn't find that in the repo.

Could you please outline the steps how to bring up a personal server or point to a wiki if already documented?

Thanks!

[jeremyletang]

Hey @choudhary001, we moved the daemon into it own repository you can find it here: github.com/ok-borg/api. There is no instructions to deploy it for now, but basically you will need an instance of mysql and elasticsearch to run the api. We will write documentation for it soon.

[crufter]

@choudhary001 I am going to cook up something today!

[choudhary001]

Hi,

Can you share some commands with which i can bring build and bring up the server? I am new to go language.

Thanks!

[crufter]

@choudhary001 yeah, I will, perhaps it's easier if we dockerise it? Would that help?

[choudhary001]

I am trying to get it up on my laptop and not really familiar with docker.

[jeremyletang]

@choudhary001 So here are some step to make the api run. First you have to build and get the dependencies and build the api binary:

$ go get
$ go build

Then you can either passes arguments to the api binary or use a confiugation file (the configuration file must be located in the same directory than the binary, and must be called .borg.conf.json. Here is an example of configuration:

{
    "mysql": {
        "addr": "192.168.99.100:3307",
        "ids": "root:root"
    },
    "github": {
        "client_id": "XXXX",
        "client_secret": "XXXX"
    },
    "esaddr": "192.168.99.100:9200"
}

Where mysql is the configuration for the mysql database, (easy enough to understand I think). The github field contains your app client Id and secret, you can genereate them by creating a github app. Then esaddr field is the address and port of you elastic search database.

If you have docker installed running an elastic search database is easy as this:

$ docker run -p 9200:9200 -d elasticsearch

If you have any more question just ask, I will be happy to help. @crufter anythig to add ?

[choudhary001]

± go version go version go1.6.3 linux/amd64

± go get package github.com/cihub/seelog: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/jinzhu/gorm: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/jinzhu/gorm/dialects/mysql: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/jpillora/go-ogle-analytics: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/julienschmidt/httprouter: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/ok-borg/api/access: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/ok-borg/api/conf: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/ok-borg/api/ctxext: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/ok-borg/api/endpoints: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/ok-borg/api/sitemap: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/ok-borg/api/types: cannot download, $GOPATH not set. For more details see: go help gopath package github.com/rs/cors: cannot download, $GOPATH not set. For more details see: go help gopath package golang.org/x/net/context: cannot download, $GOPATH not set. For more details see: go help gopath package golang.org/x/oauth2: cannot download, $GOPATH not set. For more details see: go help gopath package gopkg.in/olivere/elastic.v3: cannot download, $GOPATH not set. For more details see: go help gopath

I set the GOPATH to topfolder of cloned source. I get below error

± go get
package context: unrecognized import path "context" (import path does not begin with hostname)

How do i resolve above?

[crufter]

@choudhary001 it seems like Go is not properly installed on your machine. Helping you learn Go is obviously outside the scope here, but here are two hints: move to go 1.7+ (that's where the context error is coming from) and follow this to resolve the GOPATH issues: https://golang.org/doc/code.html

We will make sure to Release the binaries themselves with instructions on how to run it in a couple of days.

Meanwhile I suggest you to have a look at Go, it is really an awesome langauge ;)

[choudhary001]

I have done a fresh install of go. I have set the GOPATH to checked out source of api.

inno@inno-pc:~/github-projects/api$ go get

The go get command seems to be stuck with no apparent progress.

[choudhary001]

Can you tell docker steps if they are relatively simpler though i would like to build it from the source.

[crufter]

@choudhary001 do go get -v and see and paste the output please.

[choudhary001]

Hi,

± go env
GOARCH="amd64"
GOBIN="/u/choudhab/api/bin"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/u/choudhab/api"
GORACE=""
GOROOT="/u/choudhab/go/go"
GOTOOLDIR="/u/choudhab/go/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/218914811.tmpdir/go-build008239711=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

± go get -v
_/u/choudhab/api
# _/u/choudhab/api
./main.go:141: cannot use getUser (type func("golang.org/x/net/context".Context, http.ResponseWriter, *http.Request, httprouter.Params)) as type func("context".Context, http.ResponseWriter, *http.Request, httprouter.Params) in argument to access.IfAuth
./main.go:146: cannot use createSnippet (type func("golang.org/x/net/context".Context, http.ResponseWriter, *http.Request, httprouter.Params)) as type func("context".Context, http.ResponseWriter, *http.Request, httprouter.Params) in argument to access.Control
./main.go:148: cannot use updateSnippet (type func("golang.org/x/net/context".Context, http.ResponseWriter, *http.Request, httprouter.Params)) as type func("context".Context, http.ResponseWriter, *http.Request, httprouter.Params) in argument to access.Control
./main.go:149: cannot use snippetWorked (type func("golang.org/x/net/context".Context, http.ResponseWriter, *http.Request, httprouter.Params)) as type func("context".Context, http.ResponseWriter, *http.Request, httprouter.Params) in argument to access.IfAuth
./main.go:153: cannot use createOrganization (type func("golang.org/x/net/context".Context, http.ResponseWriter, *http.Request, httprouter.Params)) as type func("context".Context, http.ResponseWriter, *http.Request, httprouter.Params) in argument to access.IfAuth
./main.go:154: cannot use listUserOrganizations (type func("golang.org/x/net/context".Context, http.ResponseWriter, *http.Request, httprouter.Params)) as type func("context".Context, http.ResponseWriter, *http.Request, httprouter.Params) in argument to access.IfAuth
./main.go:157: cannot use leaveOrganization (type func("golang.org/x/net/context".Context, http.ResponseWriter, *http.Request, httprouter.Params)) as type func("context".Context, http.ResponseWriter, *http.Request, httprouter.Params) in argument to access.IfAuth
./main.go:159: cannot use expelUserFromOrganization (type func("golang.org/x/net/context".Context, http.ResponseWriter, *http.Request, httprouter.Params)) as type func("context".Context, http.ResponseWriter, *http.Request, httprouter.Params) in argument to access.IfAuth
./main.go:161: cannot use grantAdminRightToUser (type func("golang.org/x/net/context".Context, http.ResponseWriter, *http.Request, httprouter.Params)) as type func("context".Context, http.ResponseWriter, *http.Request, httprouter.Params) in argument to access.IfAuth
./main.go:165: cannot use createOrganizationJoinLink (type func("golang.org/x/net/context".Context, http.ResponseWriter, *http.Request, httprouter.Params)) as type func("context".Context, http.ResponseWriter, *http.Request, httprouter.Params) in argument to access.IfAuth
./main.go:165: too many errors

go get -v was able to download packages but it gave above errors and all subsequent runs give these errors

I tried go build which succeeded and created a binary 'api' in top dir. I was able to run the server (after running elasticsearch and mysql server with database named 'borg').

I didn't understand the part github and client_{id, secret} part in below configuration.

{
    "mysql": {
        "addr": "192.168.99.100:3307",
        "ids": "root:root"
    },
    "github": {
        "client_id": "XXXX",
        "client_secret": "XXXX"
    },
    "esaddr": "192.168.99.100:9200"
}

I got a malformed response error on client side:

± borg -h gold44.nyc 'first' Malformed response from server

± ./api 2016 Nov 04/5:46:35.37 AM EDT (main.go) [INFO] Starting http server gold44.nyc2016 Nov 04/6:14:46.83 AM EDT (query.go) [INFO] Querying first with size '5'

Can you help me make the setup work now that i have got some progress. Many thanks!

[choudhary001]

How do i generate database tables?

./api 2016 Nov 05/2:10:35.03 AM IST (main.go) [INFO] Starting http server

(Error 1146: Table 'borg.access_tokens' doesn't exist) [2016-11-05 02:14:39]

Do i need to access_token for my local instance?

[jeremyletang]

Hey, you have to run the migrations. Run the script migrate.sh inside the mifrations folder. Update the ip and port variable insidethe script with thosefrom your mysql database.

[choudhary001]

I figured that part out but it is still not working.

./borg -s '127.0.0.1' new -d Status code: 401: borg-api: Invalid access token

Not sure what access token is required for personal instance. Is there any way i can bypass authentication? Some code that i can remove?

[crufter]

@choudhary001 you have to log in with github with the frontend to acquire an access token. Later we might support cli login but for now you need the web client. If you run your own setup you need to run your own github oauth app with your own web app server.

What do you want to use the borg server? For company private snippets we just implemented the features on the api - it needs implementation on the cli or web client front though.

Unfortunately the client are lagging behind the api features, because @jeremyletang is way too quick on the backend to implement features :D. But we work hard to be up to date on the clients.

[crufter]

Closing until further questions