Attack on HackMD instance

[jakubgs]

It appears someone is attacking our HackMD instance:

The container is restarting due to the following error:

2023-05-30T22:00:11.172Z error: #011uncaughtException: Command failed: /home/hackmd/app/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs /home/hackmd/app/node_modules/markdown-pdf/phantom/render.js /tmp/tmp-22Bm6SeMfJTyGW.html /tmp/tmp-22tnP6pqwvfbnz.pdf /home/hackmd/app /home/hackmd/app/node_modules/markdown-pdf/runnings.js /home/hackmd/app/node_modules/markdown-pdf/css/pdf.css /home/hackmd/app/node_modules/highlight.js/styles/github-gist.css A4 portrait 2cm 0 10000
This application failed to start because it could not find or load the Qt platform plugin "offscreen".
Available platform plugins are: phantom.
Reinstalling the application may fix this problem.
PhantomJS has crashed. Please read the bug reporting guide at
<http://phantomjs.org/bug-reporting.html> and file a bug report.

[jakubgs]

I made two small changes thinking it might help:

• https://github.com/status-im/infra-office/commit/f0c2a2f8 - hackmd: upgrade from 2.4.1-1 to 2.4.2-1
• https://github.com/status-im/infra-office/commit/c78a644b - todo: configure and use nginx-metrics role

The second one was just to figure out what's happening.

[jakubgs]

We can see the attack in the Cloudflare dashboard:

Specifically here:

Seems to be mostly IP 91.241.49.14.

[jakubgs]

I have added the 91.241.49.14 IP to the sshguard4 IPSet:

admin@node-01.do-ams3.todo.office:~ % sudo ipset add sshguard4 91.241.49.14
admin@node-01.do-ams3.todo.office:~ % sudo ipset list sshguard4 | grep 91.241.49.14
91.241.49.14

[jakubgs]

There is a clear spike:

Seems like the main source is UK and Turkey. Weird:

[jakubgs]

Looks like we can't block by IP on the CloudFlare firewall because they move it to a separate service:

Which of course has its own separate pricing.

[jakubgs]

But I can block it probably using WAF:

Zone-level Web Application Firewall (WAF) detects and mitigates malicious requests across all traffic under this zone.

Which does a JS challenge, which we already use for some sites:

[jakubgs]

Enabled it for https://notes.status.im/:

• https://github.com/status-im/infra-office/commit/c3fb333d - firewall.tf: enable DDOS proteciton for HackMD

Lets see what that does:

[jakubgs]

At least now the process starts normally:

admin@node-01.do-ams3.todo.office:/docker/hackmd % tail -n1 /var/log/docker/hackmd-app/docker.log
2023-05-30T22:28:01.638057+00:00 docker/hackmd-app[1220993]: 2023-05-30T22:28:01.635Z info: #011HTTP Server listening at 0.0.0.0:3001

[jakubgs]

We can see the effect on the fresh new Nginx metrics for todo.office:

https://grafana.infra.status.im/d/t8QrH15Gk/nginx-metrics?orgId=1&var-fleet=todo.office&from=now-1h&to=now

[jakubgs]

There is no option for upgrading Phantom.JS separately as it has been discontinued:

• https://github.com/ariya/phantomjs/issues/15344

And CodiMD - open source HackMD - that we use is almost never updated. It is what it is.

[jakubgs]

Seems fine so far:

I'm going to leave the DDOS protection using a JS challenge enabled for now, at least until people complain it's annoying: https://github.com/status-im/infra-office/blob/c3fb333df6cfabed9b88f35ed3e3a5f69d171b7b/firewall.tf#L1-L7