search

ptpb / pb

pb is a formerly-lightweight pastebin and url shortener
Other
549 stars 52 forks source link

ptpb.pw 500 Internal Server Error on uploading png #245

Closed winny- closed 5 years ago

winny- commented 5 years ago

This is the image https://u.teknik.io/0E2nn.png .

$ curl -vvv -F c=@- https://ptpb.pw/ < blue-vaporwave.png
*   Trying 2600:1901:0:b294::1...
* TCP_NODELAY set
* Connected to ptpb.pw (2600:1901:0:b294::1) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=ptpb.pw
*  start date: Feb 19 13:12:59 2019 GMT
*  expire date: May 20 13:12:59 2019 GMT
*  subjectAltName: host "ptpb.pw" matched cert's "ptpb.pw"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> POST / HTTP/1.1
> Host: ptpb.pw
> User-Agent: curl/7.62.0
> Accept: */*
> Content-Length: 878740
> Content-Type: multipart/form-data; boundary=------------------------27863c45c51ccf5d
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 500 INTERNAL SERVER ERROR
< Server: openresty/1.13.6.2
< Date: Tue, 26 Feb 2019 19:51:27 GMT
< Content-Type: text/html
< Content-Length: 291
< Connection: keep-alive
< X-Varnish: 9677832
< Age: 0
< Via: 1.1 varnish (Varnish/6.0)
* HTTP error before end of send, stop sending
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
buhman commented 5 years ago

Not sure what was going on here; I was not able to observe a meaningful server-side error.

I restarted the pb container and am no longer able to reproduce this.

Sorry about that.

buhman commented 5 years ago 
➜  ~ curl -q https://u.teknik.io/0E2nn.png | curl -D- -F c=@- https://ptpb.pw                         
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Server: openresty/1.13.6.2
Date: Tue, 26 Feb 2019 21:43:27 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 233
Connection: keep-alive
Vary: Accept
Location: https://ptpb.pw/0aXq
X-Varnish: 295196
Age: 0
Via: 1.1 varnish (Varnish/6.0)
Accept-Ranges: bytes

date: 2019-02-26T21:43:27.899808+00:00
digest: 9d0a3340d2ff7f29d8f22fda10619a53edd1a5ea
long: AJ0KM0DS_38p2PIv2hBhmlPt0aXq
short: 0aXq
size: 878590
status: created
url: https://ptpb.pw/0aXq
uuid: 3fcf703f-7183-4905-809a-cfea0a0028eb
➜  ~ curl -X DELETE https://ptpb.pw/3fcf703f-7183-4905-809a-cfea0a0028eb
date: 2019-02-26T21:43:27.899000+00:00
digest: 9d0a3340d2ff7f29d8f22fda10619a53edd1a5ea
long: AJ0KM0DS_38p2PIv2hBhmlPt0aXq
short: 0aXq
size: 878590
status: deleted
url: https://ptpb.pw/0aXq
winny- commented 5 years ago

Thanks! I can confirm the issue is resolved.

winny- commented 5 years ago

It looks like the same fault has occurred again. It is pretty sporadic.

winston@snowcrash ~ $ date;echo;curl -sS https://u.winny.tech/lolwut1.webm | curl -D- -Fc=@- https://ptpb.pw/
Thu Mar  7 16:48:11 CST 2019

HTTP/1.1 100 Continue

HTTP/1.1 500 INTERNAL SERVER ERROR
Server: openresty/1.13.6.2
Date: Thu, 07 Mar 2019 22:48:12 GMT
Content-Type: text/html
Content-Length: 291
Connection: keep-alive
X-Varnish: 8382151
Age: 0
Via: 1.1 varnish (Varnish/6.0)

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>
buhman commented 5 years ago

:(

buhman commented 5 years ago

I still have completely no idea what's happening here.

I reproduced this, and attempted a label upload:

curl -D- -F c=@- https://ptpb.pw/~lolwut1234.webm < lolwut1.webm

The string lolwut1234 does not appear in the pb application logs, indicating pb never serviced this request at all.

I then restarted openresty, and only openresty:

buhman@instance-ptpb-aio ~ $ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                      NAMES
f4d7ae94229b        ptpb/openresty      "/usr/local/openrest…"   3 seconds ago       Up 1 second         0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   openresty
5d583f3111c9        ptpb/varnish        "varnishd -F -a :608…"   5 days ago          Up 5 days                                                      varnish
37fe62b92367        ptpb/pb             "uwsgi --ini pb.ini"     5 days ago          Up 5 days                                                      pb
e36df6481124        mongo:4             "docker-entrypoint.s…"   3 weeks ago         Up 3 weeks          27017/tcp                                  mongo

I was then no longer able to reproduce the issue. This is not helpful because:

buhman commented 5 years ago

I vaguely suspect this is some "resource not completely closed" issue between openresty/varnish. The good/bad news is this appears to be triggered over a large number of days. Reproducing this in a development environment might be tricky; some fun stats about the ptpb.pw deployment:

Over the most recent ~25,000 requests, at least 90% were related to wownero/monero command-and-control/executable distribution (specifically the minexmr.com operators appear to be abusing ptpb.pw for free network transit).

buhman commented 5 years ago

Top 5 pastes on ptpb.pw right now:

/Nqo-
/DXpq
/KAlo
/D8r9
/hgZI
buhman commented 5 years ago

Contributing factors:

To properly catch this, I'll need to:

It would also be convenient to generate/log/return request identifiers at all levels in the stack.

buhman commented 5 years ago

Tangentially I might also need to shut down ptpb.pw, until/unless I implement long-needed abuse mitigation features.

mia-0 commented 5 years ago

I run 0x0.st and I have also observed increasingly crafty attempts at uploading miner payloads, configuration files and so on. However I am actively filtering them, as well as deploying more aggressive firewall rules and malware scanning.

This and some idiot scanning all possible file URLs via anonymizing networks have caused well over 15 TB of monthly traffic since November 2018 (which doesn’t cost me a penny because I don’t run on shitty cloud hosting). It seems I have mitigated that for now.

Edit: After inspecting logs, I can assume that a majority of requests has been blocked by my dynamic firewall rules before it could hit the web server. Props to https://github.com/firehol/firehol for making this easy.