benibela / xidel

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
http://www.videlibri.de/xidel.html
GNU General Public License v3.0
674 stars 42 forks source link

https no longer works #68

Closed ralyodio closed 2 years ago

ralyodio commented 3 years ago
$ xidel -s 'https://www.marketwatch.com/investing/stock/AMD' -e '.'
Error:
Internet Error: -3 HTTPS connection failed after connecting to server. Some possible causes: handshake failure, mismatched HTTPS version/ciphers, invalid certificate
OpenSSL-Error: error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version
when talking to: https://www.marketwatch.com/investing/stock/AMD
benibela commented 3 years ago

It did not use to validate certificates properly. But now it does the validation correctly, but needs CA root certificates to that

Where are the CA certificates on your system?

Try it with the certificates from https://curl.haxx.se/ca/cacert.pem (put them where xidel can find them or use --ca-certificate=cacert.pem)

ralyodio commented 3 years ago

this seems like unnecessary work. why would curl work but not xidel?

benibela commented 3 years ago

That depends on how it was installed.

With the -v option curl shows where it is gets the CA certificates from.

On my system:

$ curl -v https://marketwatch.com
*   Trying 52.222.177.23:443...
* TCP_NODELAY set
* Connected to marketwatch.com (52.222.177.23) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs

If you have them in /etc/ssl/, too, Xidel should find them, too

ralyodio commented 3 years ago

CAPath is empty for me.

benibela commented 3 years ago

And CAFile?

ralyodio commented 3 years ago

same as yours.

benibela commented 3 years ago

Well, then it should work

Have you tried the cacert.pem ? Or the --no-check-certificate option

What is your OpenSSL version?

benibela commented 2 years ago

no feedback