Image Optimization Service Down

[briangruber]

Hi Guys,

I believe the image optimization service you are using (picperf.dev) is having some issues and all the images on the site are not loading. Not sure if you get the same thing on your end, but wanted to bring to your attention.

https://picperf.dev/https://laravel-news.com/images/icons/close.svg

Thank you for your hard work bringing all the great laravel news to the world.

[ericlbarnes]

Thanks, Brian! We will check on this.

[alexmacarthur]

@briangruber Thanks for sharing this - it's an issue apparently affecting very few people, but also tricky to nail down. Would you mind sharing a screenshot of the request headers used by your browser for one of the images? As well as which browser you're using? Feel free to message any information you don't wanna post here: https://macarthur.me/contact

Thanks!!

[njames]

I have been getting the same on Sonama and iOS but not Ubuntu /shrug

[alexmacarthur]

@njames Would you mind sending over a snippet of your request headers as well? I'm confused as why you're getting SSL-related errors. I'd also be curious to see what sort of information comes up when you navigate to the site and hover over the address bar icon here:

[alexmacarthur]

I've enabled HTTP Strict Transport Security on the domain and all subdomains to enforce requests to connect via my SSL certificate. Let me know if this had an impact on your end, @briangruber, @njames.

[njames]

Here is the smoking gun: So it seems like the certificate isn't getting read properly.

On my ubuntu machine on the same ip address it is totally fine and getting the picperf.dev cert.

[njames]

Safari on sonama is the same - if you can find anyone not upgraded to sonama and check with them that would be useful.

Safari can’t open the page “https://picperf.dev” because Safari can’t establish a secure connection to the server “picperf.dev”.

Cheers, Nigel

[njames]

In other news:

mac:

ping picperf.dev
PING warning-security.landing.telstra.com (203.50.142.140): 56 data bytes
64 bytes from 203.50.142.140: icmp_seq=0 ttl=55 time=5.673 ms
64 bytes from 203.50.142.140: icmp_seq=1 ttl=55 time=11.497 ms

ubuntu

~$ ping picperf.dev
PING picperf.dev (172.67.223.180) 56(84) bytes of data.
64 bytes from 172.67.223.180 (172.67.223.180): icmp_seq=1 ttl=50 time=185 ms
64 bytes from 172.67.223.180 (172.67.223.180): icmp_seq=2 ttl=50 time=200 ms

[alexmacarthur]

Thank you, @njames! Incredibly helpful, especially since I've been unable to replicate the issue on my end. I just went through and identified extra certificates configured for the domain. I've blown those away and replaced it with a fresh one. When you're able, please let me know if you're still seeing the same error.

[njames]

I am still seeing the same errors right now but I am also seeing the same date on the cert so I will just leave it for a bit to let the caches refresh themselves (and I did try private windows)

[alexmacarthur]

Sounds good. Hopefully, you end up seeing something looking more like this:

[briangruber]

I am using a MacBook Pro (Ventura 13.4.1)

I have tried this in chrome and safari.

I do not get the same warning that @njames got with the ping command PING warning-security.landing.telstra.com I am unable to view anything regarding the certificate in chrome's debugger:

Here are some command line tests.

curl -Iv https://picperf.dev
*   Trying 192.73.243.36:443...
* Connected to picperf.dev (192.73.243.36) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* LibreSSL SSL_connect: SSL_ERROR_ZERO_RETURN in connection to picperf.dev:443 
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_ZERO_RETURN in connection to picperf.dev:443 

openssl s_client -connect picperf.dev:443 -servername picperf.dev        
CONNECTED(00000006)
40061445F87F0000:error:0A0003E8:SSL routines:ssl3_read_bytes:reason(1000):ssl/record/rec_layer_s3.c:1586:SSL alert number 0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 313 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

python3 -m sslyze picperf.dev    

 CHECKING CONNECTIVITY TO SERVER(S)
 ----------------------------------

   picperf.dev:443           => ERROR: Unexpected connection error: "('error:140943E8:SSL routines:ssl3_read_bytes:reason(1000)\n',)"; discarding scan.

 SCANS COMPLETED IN 0.54809 S

[alexmacarthur]

Well.. that's disappointing. I'm at a loss for what's going on (assuming it's still occurring), and will likely need to reach out to Cloudflare for some additional help. I did update the minimum TLS version required for requests, and also enabled strict SSL mode within Cloudflare. Perhaps that might have done something helpful.

On my end, I have every indication that the certificate is configured correctly, from multiple remote services. Screenshots from a couple of them:

I don't know when you ran those commands, but maybe there's a chance that they run just before or while I was making other changes earlier tonight.

When you're available, I'd super appreciate giving it another shot. (and a huge thanks for being patient as I troubleshoot this!)

[cscharff]

This appears to be an issue with the end user's DNS resolver / application firewall.

[alexmacarthur]

This appears to be an issue with the end user's DNS resolver / application firewall.

That's looking more likely. I wonder if the fact that the domain is relatively new has anything to do with it.

@njames @briangruber I have another idea. Let me know if you can access this image directly through the worker endpoint:

https://picperf.amacarthur.workers.dev/https://laravel-news.com/images/icons/close.svg

[njames]

Yes I see a jolly big cross :)

and if I go to laravel-news.com we seem to be all good now

I think the cert change did it but picperf itself it is still unhappy with /shrug

[briangruber]

I am able to see the cross also if I go to the worker endpoint.

If I use any device on my home network (my iphone or macbook pro), picperf.dev doesn't work. If I switch to a different network, for example my personal hotspot on my phone, then it works. So, there is definitely something related to the network as @cscharff had suggested.

On the BROKEN network:

nslookup picperf.dev
Server:     192.168.86.1
Address:    192.168.86.1#53

Non-authoritative answer:
Name:   picperf.dev
Address: 192.73.243.36
Name:   picperf.dev
Address: 192.73.243.24

On the WORKING network:

nslookup picperf.dev
Server:     172.20.10.1
Address:    172.20.10.1#53

Non-authoritative answer:
Name:   picperf.dev
Address: 172.67.223.180
Name:   picperf.dev
Address: 104.21.80.145

I am using a Google Home and I've tried changing the DNS within the Google Home app to different resolvers like Cloudflare, etc. But it still resolves to the same broken IP. I don't have anything fancy setup on the network, pretty standard setup of a Google Mesh Network. Happy to continue debugging with you if you have suggestions.

[alexmacarthur]

Thx, @briangruber - can't emphasize how much I appreciate the back & forth on this.

I have another possible theory. The domain is using a .dev TLD, which was previously used for a number of staging and development environments. I wonder if there are ghosts in the network that are still configured to block or mess with DNS resolution through these domains.

As a way to test the waters a bit, I've wired up another one of my domains to trigger that worker. If you don't mind, try hitting this URL and let me know what you see:

https://try.jamcomments.com/https:/laravelnews.s3.amazonaws.com/featured-images/livewire-screencasts.png

[briangruber]

Of course, happy to help as much as I can.

The jamcomments URL works for me.

[alexmacarthur]

@briangruber @njames Awesome... that's promising. One more for you to try if you don't mind.

https://picperf.io/https://laravelnews.s3.amazonaws.com/featured-images/livewire-screencasts.png

[njames]

That url is fine.

I was going to tell you that the cert on the io domain was 👍 but it is just redirecting

Also this is what chrome is telling me about picperf.dev :

picperf.dev normally uses encryption to protect your information. When Chrome tried to connect 4to picperf.dev this time, the website sent back unusual and incorrect credentials. This may happen when an attacker is trying to pretend to be picperf.dev or a Wi-Fi sign-in screen has interrupted the connection. Your information is still secure because Chrome stopped the connection before any data was exchanged.

You cannot visit picperf.dev right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later.

[alexmacarthur]

Interesting! And very encouraging. I'm expecting to be further surprised at this point, but it's looking more & more likely that the *.dev domain was the issue.

[briangruber]

Yes, the picperf.io domain works. Sounds like something with the .dev like you said.

[alexmacarthur]

Thanks again for all the help with this, guys. I've further verified the issue was related to the domain, and have officially moved over to https://picperf.io.