search

Liblor / applied_sec_lab

Applied Security Laboratory - AS19
6 stars 1 forks source link

Add backup functionality #36

Closed Liblor closed 4 years ago

Liblor commented 4 years ago

This addresses #30.

keyctl commented 4 years ago

@Liblor Is this ready for review or should I wait for the separate private keys?

Liblor commented 4 years ago

@Liblor Is this ready for review or should I wait for the separate private keys?

I think you can review it

Miro-H commented 4 years ago

@Liblor Something I noted about the backups, if I list them with:

$ borg list borg_backup@aslbkp01:/backups/aslcert01 --rsh 'ssh -i /home/backup_client/.ssh/id_rsa_backup'
...
aslcert01-2019-11-14T22:15:04        Thu, 2019-11-14 22:15:05 [5dee44a7f5ae34c65e84a70c075ee125fd09f079bd6c1c8461b84f38b1dd27cd]
$ borg list borg_backup@aslbkp01:/backups/aslcert01::aslcert01-2019-11-14T22:15:04 --rsh 'ssh -i /home/backup_client/.ssh/id_rsa_backup'

I only see the latest backup folder (from the timestamp). Files that were in a previous backup are not listed. Just to be sure: Are they still in the backup, aka is the client really unable to delete files?

Liblor commented 4 years ago

Hm, good observation. I'll look at it again tomorrow to verify that this page fixes the issue. The thing is, the client is also allowed to execute delete commands, but the files get only hidden...

Miro-H commented 4 years ago

@Liblor: Backing up encrypted private keys should finally work. I tested make up followed by make push and it worked. Can you review my changes?

Liblor commented 4 years ago

I get the following error:

    aslans01: fatal: [asldb01]: FAILED! => {"changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --rsh=/usr/bin/ssh -S none -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null --rsync-path=sudo rsync --out-format=<<CHANGED>>%i %n%L /home/ansible/tls/certs/iMovies_asldb01_tls.crt asldb01:/etc/pki/tls/certs/", "msg": "Warning: Permanently added 'asldb01,172.17.0.31' (ECDSA) to the list of known hosts.\r\nrsync: link_stat \"/home/ansible/tls/certs/iMovies_asldb01_tls.crt\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]\n", "rc": 23}

Maybe it is because I merged the master branch?

Edit: Damn I didn't copy enough of the error, now it's lost :man_facepalming: but the name was some thing with "transfering".

Edit2: The error does not happen when I rerun ansible...

Edit3: nvm, it worked the second time

Miro-H commented 4 years ago

I get the following error:

    aslans01: fatal: [asldb01]: FAILED! => {"changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --rsh=/usr/bin/ssh -S none -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null --rsync-path=sudo rsync --out-format=<<CHANGED>>%i %n%L /home/ansible/tls/certs/iMovies_asldb01_tls.crt asldb01:/etc/pki/tls/certs/", "msg": "Warning: Permanently added 'asldb01,172.17.0.31' (ECDSA) to the list of known hosts.\r\nrsync: link_stat \"/home/ansible/tls/certs/iMovies_asldb01_tls.crt\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]\n", "rc": 23}

Maybe it is because I merged the master branch?

Edit: Damn I didn't copy enough of the error, now it's lost 🤦‍♂ but the name was some thing with "transferring".

Edit2: The error does not happen when I rerun ansible...

Edit3: nvm, it worked the second time

But were you able to find the source for that problem? Because it would be bad if the build failed every fifth time...

The core problem is here: /home/ansible/tls/certs/iMovies_asldb01_tls.crt\" failed: No such file or directory suggesting that for some reason no TLS certificate for asldb01 was generated. But as far as I know, ansible should not have non-deterministic behaviour and I haven't seen this error before your changes.

Liblor commented 4 years ago

No... I couldn't find the source... :(

btw, you changes seem to work for me. @keyctl do you also want to have a look at this?

keyctl commented 4 years ago

The core problem is here: /home/ansible/tls/certs/iMovies_asldb01_tls.crt\" failed: No such file or directory suggesting that for some reason no TLS certificate for asldb01 was generated. But as far as I know, ansible should not have non-deterministic behaviour and I haven't seen this error before your changes.

I had issues, too, finding the certificate on some hosts when I saw logging issues. Maybe some hosts don't get them delivered properly.

keyctl commented 4 years ago

@keyctl do you also want to have a look at this?

Yes, I will have a look in the morning. I had severe issues with both of my machines after upgrading to Fedora 31 today.

keyctl commented 4 years ago

Looks good. Are the centralized logs backed up by this? They reside on asllog* at /var/log/.

Liblor commented 4 years ago

Looks good. Are the centralized logs backed up by this? They reside on asllog* at /var/log/.

Yes: https://github.com/Liblor/applied_sec_lab/blob/f8ec4e7a9318222037a445090d95417eb4bba954/vagrant_share/ansible/logservers.yml#L9

keyctl commented 4 years ago

Looks good. Are the centralized logs backed up by this? They reside on asllog* at /var/log/.

Yes:

https://github.com/Liblor/applied_sec_lab/blob/f8ec4e7a9318222037a445090d95417eb4bba954/vagrant_share/ansible/logservers.yml#L9

Great. I'll spin up another instance with the current master on top, and check if things work.

Liblor commented 4 years ago

Thanks

I just realized, borg serve in https://github.com/Liblor/applied_sec_lab/blob/f8ec4e7a9318222037a445090d95417eb4bba954/vagrant_share/ansible/roles/backup-server/tasks/authorized_clients.yml#L2-L6

will probably no write to syslog. It should probably be borg serve --append-only --restrict-to-path {{ backup_path }}/{{ item }} | logger

keyctl commented 4 years ago

I get this when starting the backup on the log server.

● borgbackup.service - Borg
   Loaded: loaded (/etc/systemd/system/borgbackup.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2019-11-18 15:01:58 UTC; 1min 39s ago
  Process: 21672 ExecStart=/etc/borgbackup/run.sh (code=exited, status=2)
 Main PID: 21672 (code=exited, status=2)

Nov 18 15:01:58 asllog01 borgbackup[21676]: Creating backup.
Nov 18 15:01:58 asllog01 sudo[21681]:     root : TTY=unknown ; PWD=/ ; USER=backup_client ; COMMAND=/usr/bin/ssh -i /home/backup_client/.ssh/id_rsa_backup borg_backup@aslbkp01 borg serve ...
Nov 18 15:01:58 asllog01 sudo[21681]: pam_unix(sudo:session): session opened for user backup_client by (uid=0)
Nov 18 15:01:58 asllog01 borgbackup[21676]: Remote: borg_backup@aslbkp01: Permission denied (publickey).
Nov 18 15:01:58 asllog01 sudo[21681]: pam_unix(sudo:session): session closed for user backup_client
Nov 18 15:01:58 asllog01 borgbackup[21676]: Connection closed by remote host. Is borg working on the server?
Nov 18 15:01:58 asllog01 borgbackup[21676]: terminating with error status, rc 2
Nov 18 15:01:58 asllog01 borgbackup[21676]: Creating backup failed.
Nov 18 15:01:58 asllog01 systemd[1]: borgbackup.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 18 15:01:58 asllog01 systemd[1]: borgbackup.service: Failed with result 'exit-code'.
keyctl commented 4 years ago

Same goes for the alscert* machine.

Liblor commented 4 years ago

... I'll look at it shortly

Liblor commented 4 years ago

I get this when starting the backup on the log server.

● borgbackup.service - Borg
   Loaded: loaded (/etc/systemd/system/borgbackup.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2019-11-18 15:01:58 UTC; 1min 39s ago
  Process: 21672 ExecStart=/etc/borgbackup/run.sh (code=exited, status=2)
 Main PID: 21672 (code=exited, status=2)

Nov 18 15:01:58 asllog01 borgbackup[21676]: Creating backup.
Nov 18 15:01:58 asllog01 sudo[21681]:     root : TTY=unknown ; PWD=/ ; USER=backup_client ; COMMAND=/usr/bin/ssh -i /home/backup_client/.ssh/id_rsa_backup borg_backup@aslbkp01 borg serve ...
Nov 18 15:01:58 asllog01 sudo[21681]: pam_unix(sudo:session): session opened for user backup_client by (uid=0)
Nov 18 15:01:58 asllog01 borgbackup[21676]: Remote: borg_backup@aslbkp01: Permission denied (publickey).
Nov 18 15:01:58 asllog01 sudo[21681]: pam_unix(sudo:session): session closed for user backup_client
Nov 18 15:01:58 asllog01 borgbackup[21676]: Connection closed by remote host. Is borg working on the server?
Nov 18 15:01:58 asllog01 borgbackup[21676]: terminating with error status, rc 2
Nov 18 15:01:58 asllog01 borgbackup[21676]: Creating backup failed.
Nov 18 15:01:58 asllog01 systemd[1]: borgbackup.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 18 15:01:58 asllog01 systemd[1]: borgbackup.service: Failed with result 'exit-code'.

that appreas to be due to the hardening... I can't even run the run.sh script anymore. I think the hardening/firewall is too aggressive...

keyctl commented 4 years ago

Hm, have you found anything specific? I'm able to reach the backup servers from the log server, I think.

Liblor commented 4 years ago

No, I'm still investigating... some debug info:

debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "aslbkp01" port 22
debug2: ssh_connect_direct
debug1: Connecting to aslbkp01 [172.17.0.71] port 22.
debug1: Connection established.
debug1: identity file .ssh/id_rsa_backup type 0
debug1: identity file .ssh/id_rsa_backup-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u1
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to aslbkp01:22 as 'borg_backup'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from aslbkp01
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes256-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes256-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:X71aD2kXnMvBkuydTgNDzSGUBgh9sPuIhzXLVJRnsTM
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from aslbkp01
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from 172.17.0.71
debug1: Host 'aslbkp01' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: .ssh/id_rsa_backup RSA SHA256:1d187AKKTFbSnkxMX0NETp2rZYtAAJqPM2Fe9qVSNE0 explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: .ssh/id_rsa_backup RSA SHA256:1d187AKKTFbSnkxMX0NETp2rZYtAAJqPM2Fe9qVSNE0 explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
borg_backup@aslbkp01: Permission denied (publickey).
Liblor commented 4 years ago

What's the sshuser group for? Is it required to use ssh? https://github.com/Liblor/applied_sec_lab/blob/f65bf75248b7b59681898491447046e2519bc843/vagrant_share/ansible/hardening.yml#L5-L8

keyctl commented 4 years ago

Only users in that group can be SSH'd into. Then I guess it's the backup user not being in that group...

Liblor commented 4 years ago

Only users in that group can be SSH'd into. Then I guess it's the backup user not being in that group...

yes, it fixed it. I'll push the change shortly

Liblor commented 4 years ago

I create another clean build, if it works as expected I merge the branch or do you want to test it again too?

edit: I test it tomorrow, building takes so long...

keyctl commented 4 years ago

I create another clean build, if it works as expected I merge the branch or do you want to test it again too?

Yes I'm building it, too, currently.

keyctl commented 4 years ago

Ah I thought it may create the group if it does not exist. Maybe we should add it during hardening?

Liblor commented 4 years ago

My tests succeeded. @keyctl do you want to do any other tests?

Btw, borg doesn't seem to support server side logging (I couldn't find anything), which sucks :( that way a client could fake the logging info and it looks like the backup succeeded! The threat model of borg is not so well thought through in my opinion...

keyctl commented 4 years ago

We could investigate the repository and check when it was last written to by a host. The log itself is not of great use anyway by itself, because 1) it has to be read, 2) and it has to be confimed that what has been backed up is indeed what's needed to be backed up.

So verifying the archive seems the best choice, together with and IDS for production. But that's overkill for our system in my opinion. A sophisticated, solid backup is indeed possible with Borg, but it takes time and knowledge to implement it.

keyctl commented 4 years ago

My tests succeeded. @keyctl do you want to do any other tests?

I'm building already.

Liblor commented 4 years ago

We could investigate the repository and check when it was last written to by a host. The log itself is not of great use anyway by itself, because

1. it has to be read,

2. and it has to be confimed that what has been backed up is indeed what's needed to be backed up.

So verifying the archive seems the best choice, together with and IDS for production. But that's overkill for our system in my opinion. A sophisticated, solid backup is indeed possible with Borg, but it takes time and knowledge to implement it.

okay. Yes, that's true, I didn't think it through properly.

keyctl commented 4 years ago

My tests are successful. I receive logs on asllog* and can backup both aslcert* and asllog*.