search

sugarlabs / aslo-v3

Upcoming Software Center for SugarLabs
2 stars 3 forks source link

Urgent; aslo3-devel.sugarlabs.org is unmaintained #14

Open quozl opened 5 years ago

quozl commented 5 years ago

We received a report of aslo3-devel.sugarlabs.org using ACME TLS-SNI-1 domain validation with Let's Encrypt, which will shortly cease to work. On investigating the virtual machine aslo3 on justice, the system had not been kept up to date with security updates. There were about 200 to apply.

quozl commented 5 years ago

I've applied security updates and fixed Let's Encrypt configuration. Others may take the remaining tasks.

llaske commented 5 years ago

Guess that these machines are here for historical reason. BTW, may be it will be easier to maintain if it's hosted on a public cloud.

jatindhankhar commented 5 years ago

Thanks for applying the fix. If I remember last when I renewed certificate, certbot made a cron file to attemp auto renewals at /etc/cron.d/certbot

I have restarted the aslo3 server , so it's back up again.

sudo systemctl start aslo3-devel

Mongo server was down which was causing Gateway time out connections.

There are few changes that need to be done. I can take up some of them during the weekend.

  1. Syncing changes made during GCI to the build.

I also saw lots of ssh login attempts (bots ?) 

Jan 29 11:38:22 aslo3 sshd[28119]: Connection closed by 200.195.171.75 port 39832 [preauth]
Jan 29 11:38:23 aslo3 sshd[28120]: Connection closed by 200.195.171.74 port 43528 [preauth]
Jan 29 11:38:23 aslo3 sshd[28123]: Connection closed by 200.178.102.24 port 35666 [preauth]
Jan 29 11:38:24 aslo3 sshd[28125]: Connection closed by 200.137.2.254 port 59132 [preauth]
Jan 29 11:38:24 aslo3 sshd[28126]: Connection closed by 200.137.2.138 port 44058 [preauth]
Jan 29 11:38:25 aslo3 sshd[28129]: Connection closed by 150.165.85.5 port 52651 [preauth]
Jan 29 11:38:25 aslo3 sshd[28131]: Connection closed by 200.150.77.50 port 41221 [preauth]
Jan 29 11:38:28 aslo3 sshd[28133]: Connection closed by 200.132.36.194 port 46710 [preauth]
Jan 29 11:38:28 aslo3 sshd[28135]: Connection closed by 200.20.164.158 port 45118 [preauth]
Jan 29 11:38:28 aslo3 sshd[28134]: Connection closed by 200.132.35.25 port 41270 [preauth]
Jan 29 11:38:33 aslo3 sshd[28139]: Connection closed by 200.195.171.75 port 44210 [preauth]
Jan 29 11:38:33 aslo3 sshd[28140]: Connection closed by 200.195.171.74 port 47912 [preauth]
Jan 29 11:38:33 aslo3 sshd[28143]: Connection closed by 186.208.23.205 port 55849 [preauth]
Jan 29 11:38:35 aslo3 sshd[28145]: Connection closed by 200.178.102.24 port 40053 [preauth]
Jan 29 11:38:35 aslo3 sshd[28146]: Connection closed by 200.137.2.254 port 34220 [preauth]
Jan 29 11:38:35 aslo3 sshd[28147]: Connection closed by 200.137.2.138 port 48385 [preauth]
Jan 29 11:38:36 aslo3 sshd[28152]: Connection closed by 200.150.77.50 port 45598 [preauth]
Jan 29 11:38:36 aslo3 sshd[28151]: Connection closed by 150.165.85.5 port 57021 [preauth]
quozl commented 5 years ago

@llaske, thanks, but a server in public cloud would be little different; and the lower visibility would split the focus of the sysadmin team further.

@jatindhankhar, yes, we see those SSH brute force attempts on all our servers at Sugar Labs, and I see them on my servers at OLPC. It's not much to worry about, given that sshd_config has PasswordAuthentication set to no, but if you are worried the fail2ban software can be used to detect and block. However, fail2ban is a common cause for locking yourself out of your own server, so care is needed. :grin:

@scanterog, keeping you informed.

jatindhankhar commented 5 years ago

There is some issue. All the data from mongodb is gone. Although var/lib/mongodb has some files, not sure if they can be used to recover. activity collection have zero records

aslo3-devel-m2.activity
> db.activity.count()
0

This was in the logs at /var/log/mongodb

2019-01-22T20:36:02.500-0500 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-01-22T20:36:02.500-0500 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
quozl commented 5 years ago

Thanks, interesting. But I'm taking no further action on this. http://activities.sugarlabs.org/ has the features I need at the moment, and without these features I can't transition to aslo-v3. A list of features needed is in the September 2018 discussion on sugar-devel@, Deployment of ASLOv3.

jatindhankhar commented 5 years ago

No problem, I understand. I will try to sort it over the weekend.

On Thu 31 Jan, 2019, 2:05 AM James Cameron <notifications@github.com wrote:

Thanks, interesting. But I'm taking no further action on this. http://activities.sugarlabs.org/ has the features I need at the moment, and without these features I can't transition to aslo-v3. A list of features needed is in the September 2018 discussion on sugar-devel@, Deployment of ASLOv3 http://lists.sugarlabs.org/archive/sugar-devel/2018-September/thread.html .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sugarlabs/aslo-v3/issues/14#issuecomment-459098830, or mute the thread https://github.com/notifications/unsubscribe-auth/ACTOMVROsiaF6mirMQIkrfGGHrvGmIXaks5vIgIMgaJpZM4aXLT2 .

jatindhankhar commented 5 years ago

It's running the latest changes made during the GCI. Sadly I wasn't able to recover the data, so will have to rebuild it again. No latest backup just a old dump.