[BUG]: script throws snyk.errors.SnykHTTPError 500 for large group

Is there an existing issue for this?

[X] I have searched the existing issues

Description of the bug

We are trying to build the dashboard out of Snyk data (SCA & Container Products) and looking out for CSV extract for the entire Snyk group, After running the script for a couple of hours, It fails with a 500 error.

Steps To Reproduce

Running the following script in ADO Pipeline -

steps:
  - task: Bash@3
    env:
      SNYK_TOKEN: $(SNYK_TOKEN)
      SNYK_GROUP: $(SNYK_GROUP)
    inputs:
      targetType: "inline"
      script: |
          docker build -t snyk-issues-to-csv .
          docker run -e SNYK_TOKEN -e SNYK_GROUP -e SNYK_OUTPUT_DIR="/project/output" --volume $(pwd):/project repo-issues-csv:latest  /app/make_issues_csvs.py
    displayName: Run Snyk Data Collection Container

The script runs fine for 2 hours and then fails with 500 Response 

2023-08-30T18:24:39.2938544Z snyk.errors.SnykHTTPError: <Response [500]>
2023-08-30T18:24:39.5866437Z ##[error]Bash exited with code '1'.
2023-08-30T18:24:39.5893166Z ##[section]Finishing: Run Snyk Data Collection Container

Additional Information

Shared full log file and Snyk group ID with @nathan-roys

Attempted this 3 times, 3 out 3 times failed after approximately 2 hours run (full logs are shared with @nathan-roys) -

2023-08-31T10:51:17.4635159Z Traceback (most recent call last): 2023-08-31T10:51:17.4635411Z File "/app/make_issues_csvs.py", line 114, in 2023-08-31T10:51:17.4635918Z process_org(org, client, group_path, the_date) 2023-08-31T10:51:17.4636096Z File "/app/make_issues_csvs.py", line 19, in process_org 2023-08-31T10:51:17.4636275Z update_project_issues(p.id, client, org_id, org_path, the_date) 2023-08-31T10:51:17.4636465Z File "/app/make_issues_csvs.py", line 29, in update_project_issues 2023-08-31T10:51:17.4636655Z save_project_issues(p_id, client, org_id, project_csv, the_date) 2023-08-31T10:51:17.4636837Z File "/app/make_issues_csvs.py", line 51, in save_project_issues 2023-08-31T10:51:17.4638381Z req = client.post(f'reporting/issues/?from={the_date}&to={the_date}&page={x}&perPage={per_page}&sortBy=issueTitle&order=asc&groupBy=issue', i_filter) 2023-08-31T10:51:17.4638886Z File "/usr/local/lib/python3.9/site-packages/snyk/client.py", line 77, in post 2023-08-31T10:51:17.4639053Z resp = retry_call( 2023-08-31T10:51:17.4639304Z File "/usr/local/lib/python3.9/site-packages/retry/api.py", line 101, in retry_call 2023-08-31T10:51:17.4639550Z return retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter, logger) 2023-08-31T10:51:17.4639884Z File "/usr/local/lib/python3.9/site-packages/retry/api.py", line 33, in retry_internal 2023-08-31T10:51:17.4640048Z return f() 2023-08-31T10:51:17.4640286Z File "/usr/local/lib/python3.9/site-packages/snyk/client.py", line 71, in request 2023-08-31T10:51:17.4649052Z raise SnykHTTPError(resp) 2023-08-31T10:51:17.4649334Z snyk.errors.SnykHTTPError: <Response [500]> 2023-08-31T10:51:17.6999158Z ##[error]Bash exited with code '1'. 2023-08-31T10:51:17.7024034Z ##[section]Finishing: Run Snyk Data Collection Container

Screenshot 2023-08-31 at 12 00 47

snyk-labs / snyk-issues-to-csv