cubiclesoft
commented
1 year ago According to SSL Labs, that web server is presenting an incomplete certificate chain:
This server's certificate chain is incomplete. Grade capped to B.
So what is happening is that your web browser sees an incomplete certificate chain and downloads the missing intermediate and then validates the certificate chain to the root cert. This library, cURL, and PHP itself won't do that. They actually have a misconfigured web server, which might be intentional (i.e. to prevent web scrapers from functioning).
The quick fix is to disable peer verification of the certificate chain:
<?php
$rootpath = str_replace("\\", "/", dirname(__FILE__));
require_once $rootpath . "/support/http.php";
require_once $rootpath . "/support/web_browser.php";
$url = "https://www.tdcj.texas.gov/";
$web = new WebBrowser();
$options = array(
"sslopts" => HTTP::GetSafeSSLOpts(),
"debug" => true
);
// This is, in general, a bad idea.
$options["sslopts"]["verify_peer"] = false;
$result = $web->Process($url, $options);
var_dump($result);
?>To fix this the "right way," you need to download both the root and the intermediate certificates for the domain as a single PEM file and then reference the PEM file in your code.
I've uploaded the appropriate file here as exported from Firefox: tdcj-texas-gov-chain.zip
<?php
$rootpath = str_replace("\\", "/", dirname(__FILE__));
require_once $rootpath . "/support/http.php";
require_once $rootpath . "/support/web_browser.php";
$url = "https://www.tdcj.texas.gov/";
$web = new WebBrowser();
$options = array(
"sslopts" => HTTP::GetSafeSSLOpts($rootpath . "/tdcj-texas-gov-chain.pem"),
"debug" => true
);
$result = $web->Process($url, $options);
var_dump($result);
?>I was stumped for a while on this one. Diagnosing SSL/TLS issues is difficult. Turning on debugging will usually dump some useful error output. SSL Labs is useful for certain problems. But it is safe to say that it is the fault of whoever is managing that system. Law enforcement and judicial entities are notorious for not being particularly transparent. They put up digital walls to block access to important information. In most cases, those digital walls actually directly violate the written law. This could simply be a server misconfiguration - it happens from time to time - but given the tendencies of judicial entities, it is just as likely an attempt to be intentionally malicious to outside entities, especially media outlets.
You can do similar things with cURL (disable peer verification, use alternate certificate chains, etc). I prefer my own toolkit as it is the Swiss army knife of web scraping software and implements the HTTP protocol directly with lots of debugging support. PHP's built-in facilities for diagnosing SSL/TLS issues are a bit on the weak side of things regardless of what you use.
syltrinket
At my organization, we work a lot with prisoners in Texas, and because they are transferred fairly frequently, it helps to be able to click a button to retrieve their current location within our contract tracking software we use.
I used to do this with cURL, but something changed at the site that is disallowing a connection. This is an example of a random person's page: https://inmate.tdcj.texas.gov/InmateSearch/viewDetail.action?sid=08195689. We just grab the "current facility" name. However, now I can't even access the main page here with cURL: https://www.tdcj.texas.gov/.
I'm using PHP 7.4, and this is the code I am trying with ultimate-web-scraper, from the examples:
The response I get is: "Error retrieving URL. Unable to retrieve content. Unable to establish a connection to 'ssl://www.tdcj.texas.gov:443'."
I'm sorry if this is a really basic question, but I'm not very familiar with scraping and how to address problems like this. I have little use for collecting masses of data, just small needs to help make things easier for our volunteers.
thanks for any pointers you might be able to provide!