JSON output from scan is non-compliant

[wizdude]

i've been using tls-scan to perform an audit of certificates on my local network. the output from the scan is meant to be in JSON format, but it appears to be non-compliant.

if i use https://jsonlint.com/ or https://jsonformatter.org/ i get parsing errors. same problem if i try to use JSON import into Excel.

if there is another product or something i can use to view or convert the results i'm happy to use this. otherwise the output may need to be changed.

at this stage, i'd even be happy for a rex/sed/shell hack of any kind to convert the output.

does anyone have any ideas?

my goal is simply to be able to get the scanned data into a nice readable format of some kind. i need to be able to see the IP address, the port number, the certificate which was found and the related certificate dates. the other data would be nice as well, but not essential.

many thanks in advance.

[prbinu]

Hi @wizdude , please share the offending json output along with the full tls-scan command line.

[wizdude]

greetings,

command line: ./tls-scan --infile=network-192.168.xx-mini --outfile=scan-192.168.xx-mini --timeout=1

the file network-192.168.xx-mini contains an IP and a port on each line, for example:

192.168.20.30:443
192.168.20.30:444
192.168.20.30:7443
192.168.20.30:8443
192.168.20.30:9443

the output is:

{ "host": "192.168.20.40", "ip": "192.168.20.40", "port": 443, "elapsedTime": 16, "tlsVersion": "TLSv1.2", "cipher": "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD", "tempPublicKeyAlg": "ECDH secp384r1", "tempPublicKeySize": 384, "secureRenego": true, "compression": "NONE", "expansion": "NONE", "x509ChainDepth": 2, "verifyCertResult": false, "verifyCertError": "unable to get local issuer certificate", "verifyHostResult": false, "ocspStapled": true, "verifyOcspResult": false, "certificateChain": [ { "version": 3, "subject": "CN=xxx.blah.com.au", "issuer": "CN=R3; O=Let's Encrypt; C=US", "subjectCN": "xxx.blah.com.au", "subjectAltName": "DNS:yyy.blah.com.au, DNS:xxx.blah.com.au", "signatureAlg": "sha256WithRSAEncryption", "notBefore": "Feb 26 06:06:06 2023 GMT", "notAfter": "May 27 06:06:05 2023 GMT", "expired": false, "serialNo": "aa:bb:cc:dd:ee:ff:aa:bb:cc:dd:ee:ff:aa:bb:cc:dd:ee:ff", "keyUsage": "Digital Signature, Key Encipherment critical", "extKeyUsage": "TLS Web Server Authentication, TLS Web Client Authentication", "publicKeyAlg": "RSA", "publicKeySize": 2048, "basicConstraints": "CA:FALSE critical", "subjectKeyIdentifier": "(redacted)", "sha1Fingerprint": "(redacted)" },{ "version": 3, "subject": "CN=R3; O=Let's Encrypt; C=US", "issuer": "CN=ISRG Root X1; O=Internet Security Research Group; C=US", "subjectCN": "R3", "signatureAlg": "sha256WithRSAEncryption", "notBefore": "Sep  4 00:00:00 2020 GMT", "notAfter": "Sep 15 16:00:00 2025 GMT", "expired": false, "serialNo": "(redacted)", "keyUsage": "Digital Signature, Certificate Sign, CRL Sign critical", "extKeyUsage": "TLS Web Client Authentication, TLS Web Server Authentication", "publicKeyAlg": "RSA", "publicKeySize": 2048, "basicConstraints": "CA:TRUE, pathlen:0 critical", "subjectKeyIdentifier": "(redacted)", "sha1Fingerprint": "(redacted)" } ] }
{ "host": "192.168.20.40", "ip": "192.168.20.40", "port": 444, "elapsedTime": 6, "tlsVersion": "TLSv1.2", "cipher": "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD", "tempPublicKeyAlg": "ECDH secp384r1", "tempPublicKeySize": 384, "secureRenego": true, "compression": "NONE", "expansion": "NONE", "x509ChainDepth": 1, "verifyCertResult": false, "verifyCertError": "unable to get local issuer certificate", "verifyHostResult": false, "ocspStapled": false, "certificateChain": [ { "version": 3, "subject": "XXXX-123", "issuer": "CN=XXXX-123", "subjectCN": "XXXX-123", "subjectAltName": "DNS:XXXX-123, DNS:XXXX-123.blah.biz", "signatureAlg": "sha1WithRSAEncryption", "notBefore": "Jul 21 05:37:28 2021 GMT", "notAfter": "Jul 21 05:37:28 2026 GMT", "expired": false, "serialNo": "(redacted)", "keyUsage": "Digital Signature, Key Encipherment critical", "extKeyUsage": "TLS Web Server Authentication", "publicKeyAlg": "RSA", "publicKeySize": 2048, "basicConstraints": "CA:FALSE critical", "sha1Fingerprint": "(redacted)" } ] }

many thanks

[prbinu]

Your output appears to be working fine with jq:

$ cat tls-scan.out | jq
{
  "host": "192.168.20.40",
  "ip": "192.168.20.40",
  "port": 443,
  "elapsedTime": 16,
  "tlsVersion": "TLSv1.2",
  "cipher": "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD",
  "tempPublicKeyAlg": "ECDH secp384r1",
  "tempPublicKeySize": 384,
  "secureRenego": true,
  "compression": "NONE",
  "expansion": "NONE",
  "x509ChainDepth": 2,
  "verifyCertResult": false,
  "verifyCertError": "unable to get local issuer certificate",
  "verifyHostResult": false,
  "ocspStapled": true,
  "verifyOcspResult": false,
  "certificateChain": [
    {
      "version": 3,
      "subject": "CN=xxx.blah.com.au",
      "issuer": "CN=R3; O=Let's Encrypt; C=US",
      "subjectCN": "xxx.blah.com.au",
      "subjectAltName": "DNS:yyy.blah.com.au, DNS:xxx.blah.com.au",
      "signatureAlg": "sha256WithRSAEncryption",
      "notBefore": "Feb 26 06:06:06 2023 GMT",
      "notAfter": "May 27 06:06:05 2023 GMT",
      "expired": false,
      "serialNo": "aa:bb:cc:dd:ee:ff:aa:bb:cc:dd:ee:ff:aa:bb:cc:dd:ee:ff",
      "keyUsage": "Digital Signature, Key Encipherment critical",
      "extKeyUsage": "TLS Web Server Authentication, TLS Web Client Authentication",
      "publicKeyAlg": "RSA",
      "publicKeySize": 2048,
      "basicConstraints": "CA:FALSE critical",
      "subjectKeyIdentifier": "(redacted)",
      "sha1Fingerprint": "(redacted)"
    },
    {
      "version": 3,
      "subject": "CN=R3; O=Let's Encrypt; C=US",
      "issuer": "CN=ISRG Root X1; O=Internet Security Research Group; C=US",
      "subjectCN": "R3",
      "signatureAlg": "sha256WithRSAEncryption",
      "notBefore": "Sep  4 00:00:00 2020 GMT",
      "notAfter": "Sep 15 16:00:00 2025 GMT",
      "expired": false,
      "serialNo": "(redacted)",
      "keyUsage": "Digital Signature, Certificate Sign, CRL Sign critical",
      "extKeyUsage": "TLS Web Client Authentication, TLS Web Server Authentication",
      "publicKeyAlg": "RSA",
      "publicKeySize": 2048,
      "basicConstraints": "CA:TRUE, pathlen:0 critical",
      "subjectKeyIdentifier": "(redacted)",
      "sha1Fingerprint": "(redacted)"
    }
  ]
}
{
  "host": "192.168.20.40",
  "ip": "192.168.20.40",
  "port": 444,
  "elapsedTime": 6,
  "tlsVersion": "TLSv1.2",
  "cipher": "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD",
  "tempPublicKeyAlg": "ECDH secp384r1",
  "tempPublicKeySize": 384,
  "secureRenego": true,
  "compression": "NONE",
  "expansion": "NONE",
  "x509ChainDepth": 1,
  "verifyCertResult": false,
  "verifyCertError": "unable to get local issuer certificate",
  "verifyHostResult": false,
  "ocspStapled": false,
  "certificateChain": [
    {
      "version": 3,
      "subject": "XXXX-123",
      "issuer": "CN=XXXX-123",
      "subjectCN": "XXXX-123",
      "subjectAltName": "DNS:XXXX-123, DNS:XXXX-123.blah.biz",
      "signatureAlg": "sha1WithRSAEncryption",
      "notBefore": "Jul 21 05:37:28 2021 GMT",
      "notAfter": "Jul 21 05:37:28 2026 GMT",
      "expired": false,
      "serialNo": "(redacted)",
      "keyUsage": "Digital Signature, Key Encipherment critical",
      "extKeyUsage": "TLS Web Server Authentication",
      "publicKeyAlg": "RSA",
      "publicKeySize": 2048,
      "basicConstraints": "CA:FALSE critical",
      "sha1Fingerprint": "(redacted)"
    }
  ]
}

[wizdude]

while this passes in jq, i couldn't get it to work anywhere else.

i managed to resolve the problem by implementing a jq filter to output the required data with each certificate on a separate line.

cat scan-192.168.20 \
| jq -r '"\(.ip)|\(.port)|\(.tlsVersion)|\(.certificateChain[] | {subject,subjectCN,subjectAltName,notBefore,notAfter,expired} | join("|"))"' \
> output-192.168.20

i had to use a pipe character as a delimiter because many of the subject names contain commas and it was much easier to do it this way rather than trying to reformat the text fields to make them compliant.

the end result flattens each certificate to a single line, so if multiple certs or a chain come up for a host it creates separate host line entries. as an added bonus, excel text import automatically understands the use of a pipe character as a delimiter so i was able to work with the data fairly easily.

hope this helps someone else. not sure if you have a handy place to post this up as part of any documentation.

edit: jqplay.org is a great site to allow you to test out filters. it made the job a lot easier.

cheers, wizdude.

[prbinu]

Hi @wizdude , I manually reviewed the json file, but nothing stood out. Could you share the offending json field?

[wizdude]

If you take the output I provided above, or even the output you provided and pass it through either https://jsonlint.com/ or https://jsonformatter.org/ you get parsing errors.

the end of the certificatechain array is where it breaks. It doesn’t like the formatting when the next host record starts.

this all started when I tried to take the output from tls-scan and import it into excel. I couldn’t get it to work. After going through some support options, it was shown to me that because the json didn’t validate it wouldn’t work. If I have json that validates clean then it imports.

I spent some time trying to understand what part of the output and its formatting was causing the problem but after burning a lot of time I decided to work around the issue by converting the output as I posted above.

It looks like the problem should be minor, but I couldn’t work out how to change the formatting to make it compliant.

If I can provide anything else to assist let me know.

Cheers.

[wizdude]

if i take the output that you provided above and put it into jsonlint, i get the following result:

Error: Parse error on line 57:
...redacted)"       }   ]} {    "host": "192.168.
--------------------^
Expecting 'EOF', '}', ',', ']', got '{'

[prbinu]

...redacted)"       }   ]} {    "host": "192.168.
--------------------^
Expecting 'EOF', '}', ',', ']', got '{'

@wizdude , the error you see is because there are two (separate) json outputs you are pasting as input, and it is expected to throw error. Try one scan output at a time, and it is working (try in incognito, as the https://jsonformatter.org/ appears to have some caching issues).