search

TruCol / Self-host-GitLab-CI-for-GitHub

Installs your own GitLab CI and runs it on all your GitHub repos, in a single command.
GNU Affero General Public License v3.0
4 stars 3 forks source link

Support https/ssl #144

Closed a-t-0 closed 1 year ago

a-t-0 commented 1 year ago

To make this work,

  1. Install Qemu.
  2. In Qemu, install GitLab.
  3. Create the self-signed SSL certificates as generated for Nextcloud here.
  4. Copy the certificates into the correct domain.
  5. Then modify the GitLab.rb as described below.
  6. Reconfigure Gitlab.
  7. Visit the local urls below inside the qemu Firefox browser to see if HTTPS, and hence SSL works.
  8. Once https/SSL works, go to next step of copy GitHub2GitLab/clone from GitLab server.

https://www.youtube.com/watch?v=2XE6axvWXSU

In ~/gitlab//config/GitLab.rb change

external_url=127.0.0.1

to:

external_url 'https:///127.0.0.1'

or:

external_url 'https:///p7f3sxi4avnpwqwra63xhdb3rfrdd6axjm35ycptqudauqaf43vamsid.onion'

Create dir:

sudo mkdir -p /etc/gitlab/ssl
sudo chmod 775 /etc/gitlab/ssl

cd /home/name/git/Hiveminds/Collabora-Online
sudo cp cert.pem /etc/gitlab/ssl/127.0.0.1.crt
sudo cp cert-key.pem /etc/gitlab/ssl/127.0.0.1.key

sudo cp cert.pem /etc/gitlab/ssl/p7f3sxi4avnpwqwra63xhdb3rfrdd6axjm35ycptqudauqaf43vamsid.onion.crt
sudo cp cert-key.pem /etc/gitlab/ssl/p7f3sxi4avnpwqwra63xhdb3rfrdd6axjm35ycptqudauqaf43vamsid.onion.key

# Based on:
# sudo cp SSL_PUBLIC_KEY_FILENAME /etc/gitlab/ssl/127.0.0.1.crt
# sudo cp SSL_PRIVATE_KEY_FILENAME /etc/gitlab/ssl/127.0.0.1.key

gitlab-ctl reconfigure.

sudo docker ps -a
e0bdc6ec75b7
sudo docker exec -i e0bdc6ec75b7 bash -c "gitlab-ctl reconfigure"
a-t-0 commented 1 year ago

! Most root CA's are included by default

nginx['ssl_client_certificate'] = "/etc/gitlab/ssl/ca.crt"

nginx['ssl_certificate'] = "/etc/gitlab/ssl/#{node['fqdn']}.crt"

nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/#{node['fqdn']}.key"

nginx['ssl_client_certificate'] = "/etc/gitlab/ssl/ca.crt" nginx['ssl_certificate'] = "/etc/gitlab/ssl/public_certificate.crt" nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/private.key"

https://stackoverflow.com/questions/19979171/how-to-convert-pem-into-key

Retry

sudo cp ca.crt /etc/gitlab/ssl/ca.crt

openssl x509 -outform der -in cert.pem -out public_certificate.crt
sudo cp public_certificate.crt /etc/gitlab/ssl/public_certificate.crt

openssl rsa -outform der -in cert-key.pem -out private.key
sudo cp private.key /etc/gitlab/ssl/private.key
a-t-0 commented 1 year ago

https://127.0.0.1:22 Nothing https://127.0.0.1:23 Nothing https://127.0.0.1:80 Nothing https://127.0.0.1:81 Nothing https://127.0.0.1:443 Nextcloud https://127.0.0.1:444 Nothing

https://localhost:22 Nothing https://localhost:23 Nothing https://localhost:80 Nothing https://localhost:81 Nothing https://localhost:443 Nextcloud https://localhost:444 Nothing

http://127.0.0.1:22 Nothing http://127.0.0.1:23 Nothing http://127.0.0.1:80 GitLab http://127.0.0.1:81 Nextcloud http://127.0.0.1:443 Bad request http://127.0.0.1:444 Nothing

http://localhost:22 Nothing http://localhost:23 Nothing http://localhost:80 GitLab http://localhost:81 Nextcloud http://localhost:443 Bad request http://localhost:444 Nothing

Bad Request Your browser sent a request that this server could not understand. Reason: You're speaking plain HTTP to an SSL-enabled server port. Instead use the HTTPS scheme to access this URL, please.

https://127.0.0.1:22 Nothing https://127.0.0.1:23 Nothing https://p7f3sxi4avnpwqwra63xhdb3rfrdd6axjm35ycptqudauqaf43vamsid.onion:80 Can't be reached https://127.0.0.1:81 Nothing https://p7f3sxi4avnpwqwra63xhdb3rfrdd6axjm35ycptqudauqaf43vamsid.onion:443 Cant provide secure connection https://127.0.0.1:444 Nothing

http://127.0.0.1:22 Nothing http://127.0.0.1:23 Nothing http://p7f3sxi4avnpwqwra63xhdb3rfrdd6axjm35ycptqudauqaf43vamsid.onion:80 Can't be reached http://127.0.0.1:81 Nothing http://p7f3sxi4avnpwqwra63xhdb3rfrdd6axjm35ycptqudauqaf43vamsid.onion:443 Cant provide secure connection http://127.0.0.1:444 Nothing

This site can’t provide a secure connectionp 7f3sxi4avnpwqwra63xhdb3rfrdd6axjm35ycptqudauqaf43vamsid.onion sent an invalid response. ERR_SSL_PROTOCOL_ERROR

a-t-0 commented 1 year ago

Source: https://www.unixmen.com/how-to-install-and-configure-qemu-in-ubuntu/

sudo apt-get install qemu-kvm qemu virt-manager virt-viewer libvirt-bin

becomes:

sudo apt-get install qemu-kvm

Then create image with:

qemu-img create ubuntu.img 20G

Becomes:

cd ~/Downloads
qemu-img create ubuntu.img 20G

To boot the Virtual machine, run:

qemu-system-x86_64 -hda ubuntu.img -boot d -cdrom /home/sk/Soft_Backup/OS\ Images/New/ubuntu-15.04-server-amd64.iso -m 640

Becomes:

qemu-system-x86_64 -hda ubuntu.img -boot d -cdrom ~/Downloads/ubuntu-22.04.1-desktop-amd64.iso -m 640

or:

cd ~/Downloads
qemu-img create ubuntu21.img 20G
qemu-system-x86_64 --enable-kvm -m 1024 -machine smm=off -cdrom ~/Downloads/ubuntu-20.04.5-desktop-amd64.iso -boot order=d ubuntu21.img

Ubuntu 22.10:

cd ~/Downloads
qemu-img create ubuntu22.img 20G
qemu-system-x86_64 --enable-kvm -m 1024 -machine smm=off -cdrom ~/Downloads/ubuntu-22.04.1-desktop-amd64.iso -boot order=d ubuntu22.img

After installation, when QEMU asks, remove installation medium and press enter, close the QEMU and run:

qemu-system-x86_64 --enable-kvm -m 1024 -machine smm=off -boot order=d ubuntu22.img

Source: https://unix.stackexchange.com/a/239727 Works.

Capture keyboard:

ctrl+left_alt+G

Update

sudo apt update -y && sudo apt upgrade -y

Original

The clean , updated image is: _clean.

Copy 0

Contains failed clean installation of GitLab.

Make qemu run faster

Source add:

-smp 4

(I think to allocate 4 cpu recourses iso 1). Not sure if was actually faster or placebo.

TL;DR

Prerequisites on guest:

sudo apt install spice-vdagent

Preqequistites on host:

TODO

First line is to launch system, 2nd to 4th line are to enable copy paste with ctrl+c, ctrl+v.

qemu-system-x86_64 --enable-kvm -m 4096 -machine smm=off -boot order=d ubuntu22_clean1.img -smp 4
  -chardev qemu-vdagent,id=ch1,name=vdagent,clipboard=on \
  -device virtio-serial-pci \
  -device virtserialport,chardev=ch1,id=ch1,name=com.redhat.spice.0
a-t-0 commented 1 year ago

Allow copy paste in qemu:

qemu-system-x86_64 [ ... ] \
  -chardev qemu-vdagent,id=ch1,name=vdagent,clipboard=on \
  -device virtio-serial-pci \
  -device virtserialport,chardev=ch1,id=ch1,name=com.redhat.spice.0

qemu-system-x86_64 --enable-kvm -m 4096 -machine smm=off -boot order=d ubuntu22_clean1.img -smp 4 \
  -chardev qemu-vdagent,id=ch1,name=vdagent,clipboard=on \
  -device virtio-serial-pci \
  -device virtserialport,chardev=ch1,id=ch1,name=com.redhat.spice.0

Source: https://askubuntu.com/a/1378522

a-t-0 commented 1 year ago
  1. Copy a clean ubuntu image.
  2. Clone this repo.
  3. Make sure Gitlab runs without having to do anything without GitHub.
  4. Then do SSL.

New commands Case Sensitive

Without GitLab runner:

sudo apt install git -y
git clone https://github.com/TruCol/Self-host-GitLab-CI-for-GitHub.git
cd Sel*
git checkout separate-hub-lab
./install_gitlab.sh -s -le somegitlab@email.com -lp -lpre

With runner:

./install_gitlab -s -r -le somegitlab@email.com -lp -lpre
a-t-0 commented 1 year ago

SSL Generation Retry

Run:

cd ~
git clone https://github.com/HiveMinds/Collabora-Online.git
cd Coll*
sudo apt install tor -y

Issue:

Setting up ssl cert requires the tor domains to be created in /var/lib/tor/nextcloud/hostname. (That seems to also require one to install nextcloud).

src/./main.sh -ct
cat tor_log.txt

If that says "Bind error cannot bind to 9050... tor already running?" then kill tor:

kill -9 `pidof tor`

and try again:

src/./main.sh -ct

That hangs, and then forcefully terminate it. Then run:

src/./main.sh -h

And it should recognise the tor domain at /var/tor/nextcloud...

That also gives quite a few errors, mainly because the script also tells Nextcloud to switch from http to https, even though nextcloud is not installed.

a-t-0 commented 1 year ago

Then changing gitlab.rb and then copying the files and reconfiguring gitlab results in "letsencrypt does not give out cert for 127.0.0.1 (IP).

Solution:

  1. Separate the generate ssl creation process from nextcloud into clean version.
  2. Write script to copy certs.
  3. Write script to inject external url.
  4. Run complete steps automatically and ask feedback.
a-t-0 commented 1 year ago

Retry V3

First setup the self-hosted ssl certificates (for port 80) on a clean Ubuntu image, then install Gitlab, then verify you can visit gitlab at: https:127.0.0.1/80.

GitLab

sudo apt install git -y
mkdir git
cd git
git clone https://github.com/TruCol/Self-host-GitLab-CI-for-GitHub.git
cd Sel*
git checkout https
./install_gitlab.sh -s -le somegitlab@email.com -lp -lpre

SSL

git clone https://github.com/HiveMinds/SSL4Tor.git
cd SSL4Tor
./src/main.sh \
  --1-domain-1-service \
  --delete-onion-domain \
  --services 80:gitlab:443/22:ssh:22 \
  --make-onion-domains \
  --ssl-password somepassword \
  --background-dash \
  --make-ssl-certs \
  --setup-ssh-server \
  --get-onion-domain

Add self-signed SSL cert into GitLab

First add root CA, and then reconfigure, then see if https://127.0.0.1:80 is available.

./src/main.sh -1d1s --services 80:gitlab:443 --apply-certs \
  --convert-pem-to-crt-and-key \
  --add-root-to-gitlab

Results: Check https://localhost:80 Check https://localhost:443 Check https://localhost:8050 on a new window in firefox for:

--services 80:gitlab:80/22:ssh:22 \
--services 80:gitlab:443/22:ssh:22 \  all 4 options fail (with/without gitlabSSLcertConversion, with/without adding rootca )
--services 80:gitlab:8050/22:ssh:22 \

--services 443:gitlab:80/22:ssh:22 \
--services 443:gitlab:443/22:ssh:22 \
--services 443:gitlab:8050/22:ssh:22 \

--services 8050:gitlab:80/22:ssh:22 \
--services 8050:gitlab:443/22:ssh:22 \
--services 8050:gitlab:8050/22:ssh:22 \

For those respective GitLab port settings, for various NGINX settings.

  echo "nginx['enable'] = true" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['redirect_http_to_https'] = true" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['ssl_certificate'] = \"$ssl_public_key_in_gitlab_filepath\"" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['ssl_certificate_key'] = \"$ssl_private_key_in_gitlab_filepath\"" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  #echo "nginx['ssl_dhparam'] = \"/etc/gitlab/ssl/dhparams.pem\""  >> "$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['listen_port'] = 80" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['listen_https'] = false" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['ssl_client_certificate'] = \"/etc/gitlab/ssl/ca.crt\"" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
a-t-0 commented 1 year ago

Youtube video above says: says: 05:29 sudo cat /etc/gitlab/gitlab.rb | grep external_url

Should return the external url.

Intermediate

Convert public .pem into public .crt with:

openssl x509 -outform der -in your-cert.pem -out your-cert.crt

Convert private .pem into private .key with:

openssl pkey -in cert.pem -out cert.key

Step 3 Create /etc/gitlab/ssl directory. Because that is where it looks for these files. You can verify GitLab looks at this directory becuase the /etc/gitlab/gitlab.rb file should contain:

nginx['ssl_certificate'] = "/etc/gitlab/ssl/#{node[ 'fqdn' ]}.crt"

nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/#{node[ 'fqdn' ]}.key"

The name of the .key and .crt files should be that of the ("FULLY QUALIFIED DOMAIN NAME") = localhost or 127.0.0.1 or asdflaksjdfasdflkj.onion

So output the .key and .crt files to either: /etc/gitlab/ssl/localhost.key /etc/gitlab/ssl/localhost.crt or /etc/gitlab/ssl/127.0.0.1.key /etc/gitlab/ssl/127.0.0.1.crt or /etc/gitlab/ssl/some_long_onion.onion.key /etc/gitlab/ssl/some_long_onion.onion.crt

07:58: sudo mkdir -p /etc/gitlab/ssl sudo chmod 755 /etc/gitlab/ssl sudo cp your_cert.crt /etc/gitlab/ssl/.crt sudo cp your_key.key /etc/gitlab/ssl/.key

sudo gitlab-ctl reconfigure

a-t-0 commented 1 year ago

Below is the code I used to successfully run a self-hosted instance of GitLab over https://localhost:443 using self-signed SSL certificates:

GitLab

The GitLab instance can be installed with:

sudo apt install git -y
mkdir git
cd git
git clone https://github.com/TruCol/Self-host-GitLab-CI-for-GitHub.git
cd Sel*
git checkout https
./install_gitlab.sh \
 --server \
 --gitlab-email somegitlab@email.com \
 --gitlab-password \
 --labprereq \
 --external-url "https://0.0.0.0" \
 --gitlab-server "0.0.0.0"

SSL

The self-signed SSL certificates for the onion domains (and localhost) for GitLab can be created with: 

git clone https://github.com/HiveMinds/SSL4Tor.git
cd SSL4Tor
./src/main.sh \
  --1-domain-1-service \
  --delete-onion-domain \
  --services 443:gitlab:443/22:ssh:22 \
  --make-onion-domains \
  --ssl-password somepassword \
  --background-dash \
  --make-ssl-certs \
  --setup-ssh-server \
  --get-onion-domain

Add self-signed SSL cert into GitLab

Next, those SSL-certificates can be automatically added to GitLab, and GitLab is automatically reconfigured with:

./src/main.sh -1d1s --services 80:gitlab:443 --apply-certs

Next, one should be able to visit GitLab at https://localhost:443