Back up all my Open Source and private code repos from GitHub

[geerlingguy]

I would like to also make sure I back up all my code from GitHub, and not just rely on local clones on my workstations to be backed up via Time Machine.

I asked about this on Twitter and @cooperspencer responded with gickup, a Go-based utility that will run through an entire account or organization on GitHub, and clone or sync every single repo to a local directory (or to other software).

It's a pretty neat project, and my idea is to sync all my GitHub stuff (my personal account any my org @midwesternmac) down to my NAS, then maybe also back that folder up to my S3 Glacier-backed directory via rclone. Maybe. Having it all on GitHub + my workstation + NAS/gickup would be enough to satisfy my 3-2-1 plan, so might not also copy it all back into S3.

[geerlingguy]

Just created a new geerlingguy.go role: https://github.com/geerlingguy/ansible-role-go

[geerlingguy]

Well, it's mostly working:

pi@backup:~ $ gickup .gickup.yml 
5:13PM INF Reading /home/pi/.gickup.yml file=/home/pi/.gickup.yml
5:13PM INF grabbing the repositories from geerlingguy stage=github url=https://github.com
5:13PM INF starting backup for https://github.com/geerlingguy/3d-printing.git stage=backup
5:13PM INF cloning 3d-printing path=/Volumes/Git-Backups stage=locally
5:14PM INF starting backup for https://github.com/geerlingguy/50k-k8s-jobs.git stage=backup
5:14PM INF cloning 50k-k8s-jobs path=/Volumes/Git-Backups stage=locally
5:14PM INF starting backup for https://github.com/geerlingguy/acquia-cloud-api-scripts.git stage=backup
5:14PM INF cloning acquia-cloud-api-scripts path=/Volumes/Git-Backups stage=locally
5:14PM INF starting backup for https://github.com/geerlingguy/acquia-cloud-vm.git stage=backup
5:14PM INF cloning acquia-cloud-vm path=/Volumes/Git-Backups stage=locally
5:14PM INF starting backup for https://github.com/geerlingguy/airgradient-prometheus.git stage=backup
5:14PM INF cloning airgradient-prometheus path=/Volumes/Git-Backups stage=locally
5:14PM INF starting backup for https://github.com/geerlingguy/Android-Map-Marker-Drawables.git stage=backup
5:14PM INF cloning Android-Map-Marker-Drawables path=/Volumes/Git-Backups stage=locally
5:14PM INF starting backup for https://github.com/geerlingguy/ansible.git stage=backup
5:14PM INF cloning ansible path=/Volumes/Git-Backups stage=locally
5:17PM WRN retry 1 from 5 path=/Volumes/Git-Backups stage=locally
5:17PM INF cloning ansible path=/Volumes/Git-Backups stage=locally
5:19PM WRN retry 2 from 5 path=/Volumes/Git-Backups stage=locally
5:19PM INF cloning ansible path=/Volumes/Git-Backups stage=locally

Created issue: https://github.com/cooperspencer/gickup/issues/14

[geerlingguy]

Hmm, for ansible, I'm getting:

5:25PM INF cloning ansible path=/Volumes/Git-Backups stage=locally
5:28PM PNC symlink ../lib/ansible/cli/scripts/ansible_cli_stub.py ansible/bin/ansible: operation not supported path=/Volumes/Git-Backups stage=locally
panic: symlink ../lib/ansible/cli/scripts/ansible_cli_stub.py ansible/bin/ansible: operation not supported

[geerlingguy]

Temporary fix: Deleted Ansible fork from my account ;)

Hopefully no other repos have symlinks inside :/

[geerlingguy]

Seems like a lot of Ansible repos (and Acquia BLT, go figure) have symlinks...

6:42PM INF cloning community.digitalocean path=/Volumes/Git-Backups stage=locally
6:42PM PNC symlink digital_ocean_account_info.py community.digitalocean/plugins/modules/digital_ocean_account_facts.py: operation not supported path=/Volumes/Git-Backups stage=locally
panic: symlink digital_ocean_account_info.py community.digitalocean/plugins/modules/digital_ocean_account_facts.py: operation not supported

I'm going to set git config --global core.symlinks false since it seems they won't work on a CIFS/SMB share, then see if it clones.

[geerlingguy]

That didn't fix it.

[geerlingguy]

pi@backup:/Volumes/Git-Backups $ touch a.txt
pi@backup:/Volumes/Git-Backups $ ln -s a.txt b.sl
ln: failed to create symbolic link 'b.sl': Operation not supported

But... after finding https://superuser.com/a/1338021, I could get it working by adding mfsymlinks to the mount options!

[geerlingguy]

Blog post so I don't forget: Making sure symlinks work on CIFS/SMB mounted shares.

[geerlingguy]

Yay, got the whole thing done—though I just noticed only my public repositories were synced.

A couple other things I'd like to do:

1. Delete repositories that aren't on GitHub anymore (how?)
2. Figure out how to also clone my private repositories. (see How do I also clone private repositories?)

[geerlingguy]

Time for a syncup:

pi@backup:~ $ time gickup .gickup.yml 
...
real    3m8.268s
user    0m8.095s
sys 0m10.988s

[geerlingguy]

Seems like this is all but figured out, woohoo!