search

MihaiBojin / waf-downloader

Web Application Firewall log downloader
Apache License 2.0
0 stars 0 forks source link

Handle 5xx errors from the Cloudflare API #104

Open MihaiBojin opened 1 week ago

MihaiBojin commented 1 week ago

At times the CF API returns with 520. Handle this gracefully.

Downloading logs between 2024-11-12 20:07:00+00:00 and 2024-11-12 20:08:00+00:00 (query: get_firewall_events_ext)
Traceback (most recent call last):
  File "/usr/local/bin/waf-downloader", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/usr/local/lib/python3.13/site-packages/waf_logs_cli/cli.py", line 122, in main
    et = download_loop(
        sink=sink,
    ...<4 lines>...
        start_time=start_time,
    )
  File "/usr/local/lib/python3.13/site-packages/waf_logs/downloader.py", line 218, in download_loop
    result: LogResult = get_waf_logs(
                        ~~~~~~~~~~~~^
        zone_tag=zone_id,
        ^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        end_time=window_end,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/waf_logs/cloudflare_waf.py", line 82, in get_waf_logs
    raise RuntimeError(f"Error: {response.status_code}\n{response.text}")
RuntimeError: Error: 520
<!DOCTYPE html>
MihaiBojin commented 1 week ago

Not the same error, but related outcome:

  File "/usr/local/bin/waf-downloader", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/usr/local/lib/python3.13/site-packages/waf_logs_cli/cli.py", line 122, in main
    et = download_loop(
        sink=sink,
    ...<4 lines>...
        start_time=start_time,
    )
  File "/usr/local/lib/python3.13/site-packages/waf_logs/downloader.py", line 218, in download_loop
    result: LogResult = get_waf_logs(
                        ~~~~~~~~~~~~^
        zone_tag=zone_id,
        ^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        end_time=window_end,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/waf_logs/cloudflare_waf.py", line 87, in get_waf_logs
    raise RuntimeError(f"Errors were found: {data}")
RuntimeError: Errors were found: {'data': None, 'errors': [{'message': 'Internal server error', 'path': None, 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'timestamp': '2024-11-13T11:45:03.110411259Z'}}]}