scottmuc commented 6 months ago

In a previous issue (https://github.com/scottmuc/infrastructure/issues/69), I played with configuration to make my PI a potential git remote. Now I want to follow through with the next steps recorded there.

GitHub will remain the web UI for the mirrors of these repositories. GHA related workflows will continue here as well until a distinct separate build system is setup.

Tasks:

[x] Create the git user on the system volume and symlink the repositories
[x] Setup git.scottmuc.com public record
[x] Setup git.scottmuc.com LAN record
[x] Enable port forwarding for tcp/22
[x] Explore what ssh logs look like and setup alerts
[x] Update .mrconfig to treat git.scottmuc.com as the origin
[x] Update .mrconfig with a command to add github.com as github remotes and can sync
[x] Restructure repo names to remove repetitive naming (e.g.: git.scottmuc.com:scottmuc/scottmuc.github.com.git -> git.scottmuc.com:website.git

scottmuc commented 6 months ago

Enabling Port 22 Forwarding

The 125.124.184.131 is from 🇨🇳 .

[x] Add PasswordAuthentication no to /etc/ssh/sshd_config to machine configuration.
[x] Integrate fail2ban

scottmuc commented 6 months ago

Updating the .mrconfig consists of the following command:

cd
mr run git remote rename origin github
mr run git remote remove pi
(cd workspace/infrastructure && git remote add origin git@git.scottmuc.com:infrastructure.git)
(cd workspace/dns-zone-blocklist && git remote add origin git@git.scottmuc.com:dns-zone-blocklist.git)
(cd workspace/presentations && git remote add origin git@git.scottmuc.com:presentations.git)
(cd workspace/scottmuc.github.com && git remote add origin git@git.scottmuc.com:website.git)
(cd workspace/tabularasa && git remote add origin git@git.scottmuc.com:tabularasa.git)
mr run git fetch --all
mr run git push origin main
(cd workspace/scottmuc.github.com && git push origin source && git push origin master)

The above configures the remotes to what I desire. origin is my pi, and github is self-explanatory.

[x] Reconfigure branch names for my website (I think live and main should do just fine).

scottmuc commented 6 months ago

Disabled Port Forwarding

There were too many logs showing bots trying to ssh to my server. I'd prefer to get a bit protection in place before re-enabling it.

scottmuc commented 6 months ago

Keeping GitHub and git.scottmuc.com in sync

The above commit was pushed via the script that's committed in it:

~ ? mr run sync-origin-with-github
...

mr run: /home/wsl/workspace/infrastructure
Fetching github
Fetching origin
Already up to date.
Enumerating objects: 10, done.
Counting objects: 100% (10/10), done.
Delta compression using up to 12 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.32 KiB | 1.32 MiB/s, done.
Total 6 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To github.com:scottmuc/infrastructure.git
   0fcbcf0..d316d8c  main -> main

...

mr run: finished (5 ok)

scottmuc commented 6 months ago

Fail 2 Ban Install

With the default configuration I see fail2ban looks to be watching the correct thing by default.

root@raspberrypi:/var/log# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

Will let this run for the day and see if it catches anything.

I dropped the task to create email alerts. The scope of that is more around an intrusion detection strategy which I think is a whole other thing to look at. fail2ban at least gives me protection over a brute force attack. If someone has my keys and passphrase, even intrusion detection wouldn't help me there.

scottmuc commented 6 months ago

Fail 2 Ban Lacks Dependencies

2024-05-27 09:45:42,024 fail2ban.filter         [76411]: INFO    [sshd] Found 165.22.221.215 - 2024-05-27 09:45:42
2024-05-27 09:45:42,182 fail2ban.actions        [76411]: NOTICE  [sshd] Ban 165.22.221.215
2024-05-27 09:45:42,202 fail2ban.utils          [76411]: ERROR   7f8ebdd2f0 -- exec: { iptables -w -C f2b-sshd -j RETURN >/dev/null 2>&1; } || { iptables -w -N f2b-sshd || true;
 iptables -w -A f2b-sshd -j RETURN; }
for proto in $(echo 'tcp' | sed 's/,/ /g'); do
{ iptables -w -C INPUT -p $proto -m multiport --dports ssh -j f2b-sshd >/dev/null 2>&1; } || { iptables -w -I INPUT -p $proto -m multiport --dports ssh -j f2b-sshd; }
done
2024-05-27 09:45:42,203 fail2ban.utils          [76411]: ERROR   7f8ebdd2f0 -- stderr: '/bin/sh: 1: iptables: not found'
2024-05-27 09:45:42,203 fail2ban.utils          [76411]: ERROR   7f8ebdd2f0 -- stderr: '/bin/sh: 1: iptables: not found'
2024-05-27 09:45:42,203 fail2ban.utils          [76411]: ERROR   7f8ebdd2f0 -- stderr: '/bin/sh: 3: iptables: not found'
2024-05-27 09:45:42,204 fail2ban.utils          [76411]: ERROR   7f8ebdd2f0 -- returned 127
2024-05-27 09:45:42,204 fail2ban.utils          [76411]: INFO    HINT on 127: "Command not found".  Make sure that all commands in "{ iptables -w -C f2b-sshd -j RETURN >/dev/nul
l 2>&1; } || { iptables -w -N f2b-sshd || true; iptables -w -A f2b-sshd -j RETURN; }\nfor proto in $(echo 'tcp' | sed 's/,/ /g'); do\n{ iptables -w -C INPUT -p $proto -m multipo
rt --dports ssh -j f2b-sshd >/dev/null 2>&1; } || { iptables -w -I INPUT -p $proto -m multiport --dports ssh -j f2b-sshd; }\ndone" are in the PATH of fail2ban-server process (gr
ep -a PATH= /proc/`pidof -x fail2ban-server`/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session
and observe if additional informative error messages appear in the terminals.
2024-05-27 09:45:42,204 fail2ban.actions        [76411]: ERROR   Failed to execute ban jail 'sshd' action 'iptables-multiport' info 'ActionInfo({'ip': '165.22.221.215', 'family'
: 'inet4', 'fid': <function Actions.ActionInfo.<lambda> at 0x7f8f1bcf40>, 'raw-ticket': <function Actions.ActionInfo.<lambda> at 0x7f8f1bd6c0>})': Error starting action Jail('ss
hd')/iptables-multiport: 'Script error'

It's good to know it's spotting things, but the machine needs iptables to correctly ban the IP.

After installing iptables, I'm seeing successful bans:

root@raspberrypi:/var/log# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
f2b-sshd   tcp  --  anywhere             anywhere             multiport dports ssh

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain f2b-sshd (1 references)
target     prot opt source               destination
REJECT     all  --  165.22.221.215       anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere
root@raspberrypi:/var/log# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     111
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 1
   |- Total banned:     4
   `- Banned IP list:   165.22.221.215

scottmuc commented 6 months ago

Testing Remote Git Access

I performed the following from my laptop using my mobile phone hotspot.

~ ? curl ifconfig.co
80.187.85.136
~ ? ping git.scottmuc.com
PING git.scottmuc.com (80.137.78.16) 56(84) bytes of data.
^C
--- git.scottmuc.com ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1029ms

~ ? git clone git@git.scottmuc.com:tabularasa.git
Cloning into 'tabularasa'...
The authenticity of host 'git.scottmuc.com (80.137.78.16)' can't be established.
ED25519 key fingerprint is SHA256:Xhh1382rJGe3VdHPFadCz83Fk2BZv37yzKF5saOZ6dk.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:7: [hashed name]
    ~/.ssh/known_hosts:10: [hashed name]
    ~/.ssh/known_hosts:11: [hashed name]
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'git.scottmuc.com' (ED25519) to the list of known hosts.
remote: Enumerating objects: 179, done.
remote: Counting objects: 100% (179/179), done.
remote: Compressing objects: 100% (137/137), done.
remote: Total 179 (delta 34), reused 179 (delta 34), pack-reused 0
Receiving objects: 100% (179/179), 34.35 MiB | 4.12 MiB/s, done.
Resolving deltas: 100% (34/34), done.

Note how the IP from curl ifconfig.co is different than git.scottmuc.com.

scottmuc commented 6 months ago

Summary

Call it done! The sshd hardening along with fail2ban gives me some assurance that my machine is locked down enough. Thankfully, the machine is no longer a Samba host so I'm less worried about a compromise deleting all my stuff. My git repositories are mirrored with GitHub (and on all machines that have the repos cloned).

Next steps is to have a web UI.

scottmuc / infrastructure

"Productionize" Self Hosted GitHub #71

Enabling Port 22 Forwarding

Disabled Port Forwarding

Keeping GitHub and git.scottmuc.com in sync

Fail 2 Ban Install

Fail 2 Ban Lacks Dependencies

Testing Remote Git Access

Summary