search

CaliDog / Axeman

Axeman is a utility to retrieve certificates from Certificate Transparency Lists (CTLs)
MIT License
218 stars 71 forks source link

Crashes with "KeyError: 'entries'" #9

Open Fusl opened 5 years ago

Fusl commented 5 years ago

axeman pulled down a few TBs of CSVs from various CTLs and then crashed:

[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72408700-72408766.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72407530-72407596.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72410455-72410521.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72410195-72410261.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72412535-72412601.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72414160-72414226.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72415460-72415526.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72415850-72415916.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72417410-72417476.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72419880-72419946.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72419360-72419426.csv written!
[INFO:root] 2019-10-17 06:47:18,526 - Completed DigiCert Log Server 2, stored at /tmp/ct2.digicert-ct.com_log.csv!
[INFO:root] 2019-10-17 06:47:18,526 - Finished downloading and processing ct2.digicert-ct.com/log
[INFO:root] 2019-10-17 06:47:18,526 - Downloading certificates for DigiCert Yeti2019 Log
[INFO:root] 2019-10-17 06:47:30,526 - Starting processing coro and process pool
[1] Making dir...
[INFO:root] 2019-10-17 06:47:31,057 - Getting things to process...
[INFO:root] 2019-10-17 06:47:31,058 - Queue Status: Processing Queue Size:0 Downloaded blocks:50/1130984 (0.0044%)
Traceback (most recent call last):
  File "/usr/local/bin/axeman", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/axeman/core.py", line 295, in main
    loop.run_until_complete(retrieve_certificates(loop, concurrency_count=args.concurrency_count))
  File "uvloop/loop.pyx", line 1417, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.7/site-packages/axeman/core.py", line 112, in retrieve_certificates
    await download_tasks
  File "/usr/local/lib/python3.7/site-packages/axeman/core.py", line 53, in download_worker
    for index, entry in zip(range(start, end + 1), entry_list['entries']):
KeyError: 'entries'
josephnoir commented 2 years ago

I've run into this on the same line:

Traceback (most recent call last):
 File "$HOME/ctlogs/envs/bin/axeman", line 8, in <module>
  sys.exit(main())
 File "$HOME/ctlogs/envs/lib/python3.8/site-packages/axeman/core.py", line 295, in main
  loop.run_until_complete(retrieve_certificates(loop, concurrency_count=args.concurrency_count, output_directory=args.output_dir))
 File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
 File "$HOME/ctlogs/envs/lib/python3.8/site-packages/axeman/core.py", line 112, in retrieve_certificates
  await download_tasks
 File "$HOME/ctlogs/envs/lib/python3.8/site-packages/axeman/core.py", line 53, in download_worker
  for index, entry in zip(range(start, end + 1), entry_list['entries']):
TypeError: 'NoneType' object is not subscriptable