nelsonic commented 2 years ago

As noted in https://github.com/dwyl/github-backup/issues/136 we are considering using Gogs as a Backup Git/GitHub server.

This could potentially be a "core" element of our infrastructure at least from a Business Continuity perspective. So we need to have our SPIKE to set it up well-documented so it can be replicated & maintained.

Todo

[x] Install Gogs on Raspberry Pi 4. Requires: https://github.com/dwyl/learn-devops/issues/80
- [x] Play around with it and see what features it has.
[x] Deploy to Fly.io with lowest possible resources.
[x] Figure out how to secure it. (read docs!)

Context

For context: my plan with setting up a Gogs server is to piggy-back on the hard work done by the community in terms of creating a (simplified) Clone of GitHub that we can use to mirror all our projects. I feel this is a very worthwhile task and if we use Postgres as the DB for Gogs we can attempt to connect to it from Phoenix and thus we have most of the Tudo App.

nelsonic commented 2 years ago

Borrowing the homepage logo for the doc:

SimonLab commented 2 years ago

Gogs provides a REST API: https://github.com/gogs/docs-api/tree/master/Repositories

I think it's possible to create a new repository via the API (https://github.com/gogs/docs-api/tree/master/Repositories#create) however other commands (create commits, create branches) don't seem to be available or at least documented. @nelsonic I know you looked at using Gogs as a local server, did you look at the API at the same time?

nelsonic commented 2 years ago

@SimonLab yeah, I looked at the API and didn't see anything related to pushing commits/content. That will need to be done via Git which then means we need to figure out how to "stage" work-in-progress files. 💭

SimonLab commented 2 years ago

Phx + Git + Fly.io

I wanted to tests the idea of deploying a Phoenix application on Fly.io where Git is also installed along the application.

Create a simple Phoenix application:

mix phx.new --no-ecto --no-mailer --no-dashboard --no-live

Deploy this application on Fly.io:

fly launch

This will generate the required Docker files and fly.toml configuration file. It will also deploy the application.

Create a volume where the git repositories will be stored:

fly volumes create myapp_data --region lhr --size 1

This creates a new volume called myapp_data with a size of 1Gb

Volumes are, by default, created with encryption-at-rest enabled for additional protection of the data on the volume. Use --no-encryption to not encrypt the volume for improved performance at deployment and runtime.

Attach the new volume to the application:

Add this section to the fly.toml file

[mount]
  source="myapp_data" 
  destination="/data_repos"

The /data_repos is the path name where the volume will be mounted along your application.

deploy the application to make sure everything is ok:

flyctl deploy

ssh to your application console and check the volume is mounted:

flyctl ssh console

Once connected, run the ls command and you should see the data_repos folder

Install Git along the Phoenix application:

Open the Dockerfile file and update the section:

FROM ${RUNNER_IMAGE}

RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

to:

FROM ${RUNNER_IMAGE}

RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales git \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

And redeploy the application: flyctl deploy. To make sure git is installed, ssh to the console and run the git command

flyctl ssh console
git

Update the Phoenix application to create a new repository

An elixir git wrapper exists and I've been using it to test git init command line: https://hexdocs.pm/git_cli/api-reference.html

My controller create action takes a name as a parameter (from a form) and create a new repository

  def create(conn, params) do
    repo = params["name"]
    repos_path = Application.get_env(:gitest, :repos_path) # define path in config to avoid conflict in localhost and fly.io
    res = Git.init("#{repos_path}/#{repo}")
    IO.inspect(res)

    conn
    |> put_flash(:info, "repo created!")
    |> redirect(to: "/")
  end

It took me a bit of time to setup the Fly.io app and to make sure Git was installed but knowing these steps now, it can be recreated quickly.

nelsonic commented 2 years ago

During the initial setup on the Raspberry Pi, when you first run the Gogs web server:

./gogs web

You will be instructed to visit http://0.0.0.0:3000 ...

In my case I'm running the PI as a Server so I need to access the IP address on my Mac http://192.168.1.196:3000 (on the same network ...)

Use the IP address for the PI as the Application URL: http://192.168.1.196:3000/

Once configured, created a new user and repo: http://192.168.1.196:3000/nelsonic/my-awesome-repo

The Markdown Editor is functional: http://192.168.1.196:3000/nelsonic/my-awesome-repo/_edit/master/README.md

nelsonic commented 2 years ago

I've done a bunch of testing of Gogs on my local Raspberry Pi instance. ✅

Now trying to deploy a "Production" instance on Fly.io 🚀

Sadly, the Appkata: Gogs https://fly.io/docs/app-guides/git-gogs-server/ is "broken". 💔 Linked to from: /app-guides/git-gogs-server.html.erb#L7

flyctl init gives the following error:

Error: unknown command "init" for "flyctl"

Did you mean this?
    info

The source: https://github.com/fly-apps/appkata-gogs/ "has been archived" ... i.e. we cannot submit a PR to update it. 🤦‍♂️

Ran the command:

flyctl launch --name gogs-server --image gogs/gogs --org dwyl

Got the following output/error:

1 desired, 1 placed, 0 healthy, 1 unhealthy [health checks: 1 total, 1 critical]
v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to
Failed Instances

==> Failure #1

Instance
  ID            = 70063cfa
  Process       =
  Version       = 0
  Region        = lhr
  Desired       = run
  Status        = running
  Health Checks = 1 total, 1 critical
  Restarts      = 0
  Created       = 4m50s ago

Recent Events
TIMESTAMP            TYPE       MESSAGE
2022-04-22T00:20:04Z Received   Task received by client
2022-04-22T00:20:04Z Task Setup Building Task Directory
2022-04-22T00:20:13Z Started    Task started by client

Recent Logs
2022-04-22T00:25:04.000 [info] chmod: /data/ssh/*: No such file or directory
2022-04-22T00:25:04.000 [info] Unable to load host key: /data/ssh/ssh_host_rsa_key
2022-04-22T00:25:04.000 [info] Unable to load host key: /data/ssh/ssh_host_dsa_key
2022-04-22T00:25:04.000 [info] Unable to load host key: /data/ssh/ssh_host_ecdsa_key
2022-04-22T00:25:04.000 [info] Unable to load host key: /data/ssh/ssh_host_ed25519_key
2022-04-22T00:25:04.000 [info] sshd: no hostkeys available -- exiting.
2022-04-22T00:25:04.000 [info] mkdir: can't create directory '/data/': Permission denied
2022-04-22T00:25:04.000 [info] chmod: /data/git/.ssh: No such file or directory
2022-04-22T00:25:04.000 [info] ./run: ./setup: line 9: can't create /data/git/.ssh/environment: nonexistent directory
2022-04-22T00:25:04.000 [info] chmod: /data/git/.ssh/environment: No such file or directory
2022-04-22T00:25:04.000 [info] chmod: /data: No such file or directory
2022-04-22T00:25:04.000 [info] chmod: /data/gogs: No such file or directory
2022-04-22T00:25:04.000 [info] chmod: /data/git/: No such file or directory
2022-04-22T00:25:04.000 [info] 2022/04/22 00:25:04 [ WARN] Custom config "/data/gogs/conf/app.ini" not found. Ignore this warning if you're running for the first time
2022-04-22T00:25:04.000 [info] 2022/04/22 00:25:04 [FATAL] [gogs.io/gogs/internal/cmd/web.go:165 runWeb()] Failed to initialize application: init configuration: create SSH root directory: mkdir /data: permission denied
2022-04-22T00:25:05.000 [info] Saving key "/data/ssh/ssh_host_rsa_key" failed: No such file or directory
2022-04-22T00:25:05.000 [info] Saving key "/data/ssh/ssh_host_dsa_key" failed: No such file or directory
2022-04-22T00:25:05.000 [info] Saving key "/data/ssh/ssh_host_ecdsa_key" failed: No such file or directory
2022-04-22T00:25:05.000 [info] Saving key "/data/ssh/ssh_host_ed25519_key" failed: No such file or directory
2022-04-22T00:25:05.000 [info] chown: /data/ssh/*: No such file or directory
2022-04-22T00:25:05.000 [info] chmod: /data/ssh: No such file or directory
2022-04-22T00:25:05.000 [info] chmod: /data/ssh/*: No such file or directory
2022-04-22T00:25:05.000 [info] Unable to load host key: /data/ssh/ssh_host_rsa_key
2022-04-22T00:25:05.000 [info] Unable to load host key: /data/ssh/ssh_host_dsa_key
2022-04-22T00:25:05.000 [info] Unable to load host key: /data/ssh/ssh_host_ecdsa_key
2022-04-22T00:25:05.000 [info] Unable to load host key: /data/ssh/ssh_host_ed25519_key
2022-04-22T00:25:05.000 [info] sshd: no hostkeys available -- exiting.
2022-04-22T00:25:06.000 [info] Saving key "/data/ssh/ssh_host_dsa_key" failed: No such file or directory
2022-04-22T00:25:06.000 [info] Unable to load host key: /data/ssh/ssh_host_ecdsa_key
2022-04-22T00:25:06.000 [info] sshd: no hostkeys available -- exiting.
***v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1

Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error abort

Reading the Dockerfile: https://hub.docker.com/r/gogs/gogs ⏳

nelsonic commented 2 years ago

Working: https://gogs-server.fly.dev/

https://gogs-server.fly.dev/nelsonic/public-repo

nelsonic commented 2 years ago

Tried:

flyctl ssh console

got:


Error host unavailable: host was not found in DNS

nelsonic commented 2 years ago

Ongoing issue with hosts not resolving in the lhr (London) region: https://community.fly.io/t/phoenix-deploy-using-fly-launch/4858/2

nelsonic commented 2 years ago

Dropped an update on: https://community.fly.io/t/gogs-standalone-git-service-as-a-fly-example/358/2 thanking @codepope for his great guide and hopefully helping others with their setup/journey. 🤞

nelsonic commented 2 years ago

git clone ssh://git@gogs-server.fly.dev:10022/nelsonic/public-repo.git

That worked!

Cloning into 'public-repo'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (4/4), done.
Receiving objects: 100% (5/5), 6.62 KiB | 6.62 MiB/s, done.
remote: Total 5 (delta 0), reused 0 (delta 0), pack-reused 0

Update the README.md on my Mac:

git commit and git push the code:

git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 350 bytes | 350.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
To ssh://gogs-server.fly.dev:10022/nelsonic/public-repo.git
   7f92c5d..f714a64  master -> master

https://gogs-server.fly.dev/nelsonic/public-repo