KeyError: 'metadata' when running inside Docker

mike503 commented 2 years ago

I was able to run this on my host system, but inside of Docker, no matter what I do, it gives me this error. Even with invalid API credentials, same error.

$ docker run --rm -e CENSYS_API_ID=MY_ID -e CENSYS_API_SECRET=MY_SECRET christophetd/cloudflair redacted.com
[*] Retrieving Cloudflare IP ranges from https://www.cloudflare.com/ips-v4
[*] The target appears to be behind CloudFlare.
[*] Looking for certificates matching "redacted.com" using Censys
Traceback (most recent call last):
  File "/data/cloudflair.py", line 184, in <module>
    main(args.domain, args.output_file, censys_api_id, censys_api_secret)
  File "/data/cloudflair.py", line 153, in main
    hosts = find_hosts(domain, censys_api_id, censys_api_secret)
  File "/data/cloudflair.py", line 43, in find_hosts
    cert_fingerprints = censys_search.get_certificates(domain, censys_api_id, censys_api_secret)
  File "/data/censys_search.py", line 17, in get_certificates
    return set([ cert['parsed.fingerprint_sha256'] for cert in certificates_search_results ])
  File "/data/censys_search.py", line 17, in <listcomp>
    return set([ cert['parsed.fingerprint_sha256'] for cert in certificates_search_results ])
  File "/usr/lib/python3.6/site-packages/censys/base.py", line 173, in search
    pages = payload['metadata']['pages']
KeyError: 'metadata'

christophetd commented 2 years ago

Do you have this behavior only with your specific domain? If you use another domain, do you have the same issue?

mike503 commented 2 years ago

Same issue no matter what domain. Same issue with or without valid API credentials.

christophetd commented 2 years ago

I just pushed a new version of the Docker image, which was out of sync with the code. Can you run docker pull christophetd/cloudflair and try again?

mike503 commented 2 years ago

looks like it works. with invalid creds:

mike@local:~$ docker run --rm -e CENSYS_API_ID=MY_ID -e CENSYS_API_SECRET=MY_SECRET christophetd/cloudflair redacted.com
Unable to find image 'christophetd/cloudflair:latest' locally
latest: Pulling from christophetd/cloudflair
5d20c808ce19: Pull complete 
e04eb5a51705: Pull complete 
Digest: sha256:2d5b708c396e3b9e8bcee6c32389099fd5506a4b8554c2770a95c4a7e5ff4464
Status: Downloaded newer image for christophetd/cloudflair:latest
[*] Retrieving Cloudflare IP ranges from https://www.cloudflare.com/ips-v4
[*] The target appears to be behind CloudFlare.
[*] Looking for certificates matching "redacted.com" using Censys
[-] Your Censys credentials look invalid.
mike@local:~$

valid creds:

mike@local:~$ docker run --rm -e CENSYS_API_ID=VALID -e CENSYS_API_SECRET=VALID christophetd/cloudflair redacted.com
[*] Retrieving Cloudflare IP ranges from https://www.cloudflare.com/ips-v4
[*] The target appears to be behind CloudFlare.
[*] Looking for certificates matching "redacted.com" using Censys
[*] 12 certificates matching "redacted.com" found.
[*] Looking for IPv4 hosts presenting these certificates...
[*] 1 IPv4 hosts presenting a certificate issued to "redacted.com" were found.
  - 18.204.206.8

[*] Testing candidate origin servers
[*] Retrieving target homepage at https://redacted.com
[*] "https://redacted.com" redirected to "https://redacted.com/"
  - 18.204.206.8
[-] Did not find any origin server.
mike@local:~$

christophetd / CloudFlair

KeyError: 'metadata' when running inside Docker #55