search

singer-io / tap-pipedrive

A Singer.io tap for extracting data from the Pipedrive API
GNU Affero General Public License v3.0
13 stars 34 forks source link

"filters" endpoint leads to "argument of type 'NoneType' is not iterable" exception #138

Open christiankf opened 2 months ago

christiankf commented 2 months ago

Hi there since the 11-12 of June 2024 I got a consistent error when the Pipedrive integration run during extraction.

Here is the relevant part of the log:

2024-06-17 21:00:28,647Z    tap - CRITICAL argument of type 'NoneType' is not iterable
2024-06-17 21:00:28,648Z    tap - Traceback (most recent call last):
2024-06-17 21:00:28,648Z    tap -   File "tap-env/bin/tap-pipedrive", line 10, in <module>
2024-06-17 21:00:28,648Z    tap -     sys.exit(main())
2024-06-17 21:00:28,648Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 229, in wrapped
2024-06-17 21:00:28,648Z    tap -     return fnc(*args, **kwargs)
2024-06-17 21:00:28,648Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/cli.py", line 26, in main
2024-06-17 21:00:28,648Z    tap -     pipedrive_tap.do_sync(catalog)
2024-06-17 21:00:28,648Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/tap.py", line 237, in do_sync
2024-06-17 21:00:28,648Z    tap -     self.do_paginate(stream, stream_metadata)
2024-06-17 21:00:28,648Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/tap.py", line 273, in do_paginate
2024-06-17 21:00:28,648Z    tap -     stream.paginate(response)
2024-06-17 21:00:28,648Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/stream.py", line 74, in paginate
2024-06-17 21:00:28,648Z    tap -     if 'additional_data' in payload and 'pagination' in payload['additional_data']:
2024-06-17 21:00:28,648Z    tap - TypeError: argument of type 'NoneType' is not iterable

I had to disable to replicate the filters so the rest runs through properly.

I would expect the following fixes:

  1. Handle the error in this line. Looks easy to catch (except if it's broken contract that none of them there should be "None")
  2. Add error handling that makes sure that even if one table/endpoint breaks, the others are still run through normally
RushiT0122 commented 2 months ago

PR#137 fixes TypeError: argument of type 'NoneType' is not iterable.