airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.04k stars 4.11k forks source link

Source Facebook Marketing: user_tos_accepted is an object, not number in the schema #11800

Closed mdibaiee closed 2 years ago

mdibaiee commented 2 years ago
## Environment * **Airbyte version**: N/A * **OS Version / Instance**: macOS * **Deployment**: Docker * **Source Connector and version**: source-facebook-marketing@[0.2.42](https://hub.docker.com/layers/source-facebook-marketing/airbyte/source-facebook-marketing/0.2.42/images/sha256-3a6fe9f6f7bd5274de946008d6a4788bc893b0c9d1f1660c13805871c0f3029d?context=explore) * **Destination Connector and version**: N/A * **Severity**: High * **Step where error happened**: Sync Job ## Current Behavior In the current [`ad_account` schema definition](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/schemas/ad_account.json), `user_tos_accepted` is defined as `number,null`. This leads to an error when reading data: ``` Encountered an exception while reading stream ad_account TypeError: float() argument must be a string or a number, not 'dict' ``` ## Expected Behavior if you look at the examples [here](https://developers.facebook.com/docs/marketing-api/audiences/reference/custom-audience-terms-of-service) at the end of the page, it's an object: ![](https://user-images.githubusercontent.com/2807772/162215637-dfcd90d9-7409-4fd1-9c97-ee7bf5c3ac25.png) Also see Facebook Business SDK's definition of this field: * https://github.com/facebook/facebook-python-business-sdk/blob/main/facebook_business/adobjects/adaccount.py#L3964 * Which comes from this specification: https://github.com/facebook/facebook-business-sdk-codegen/blob/2c2c48483d5b33e2759aca68d7ed92dc9b1775ea/api_specs/specs/AdAccount.json#L3730-L3731 And there should be no errors. ## Logs
LOG ``` Encountered an exception while reading stream ad_account^ Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 119, in read internal_config=internal_config, File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 159, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 249, in _read_full_refresh yield self._as_airbyte_record(configured_stream.stream.name, record) File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 281, in _as_airbyte_record transformer.transform(data, schema) # type: ignore File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/utils/transform.py", line 172, in transform for e in normalizer.iter_errors(record): File "/usr/local/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/utils/transform.py", line 152, in normalizator instance[k] = self.__normalize(instance[k], subschema) File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/utils/transform.py", line 79, in __normalize original_item = self.default_convert(original_item, subschema) File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/utils/transform.py", line 110, in default_convert return float(original_item) TypeError: float() argument must be a string or a number, not 'dict'" ```
## Steps to Reproduce 1. Run an Airbyte connector on a Facebook account with Custom TOS ## Are you willing to submit a PR? Opened PR: https://github.com/airbytehq/airbyte/pull/11801 ┆Issue is synchronized with this [Monday item](https://airbyte-bunch.monday.com/boards/2536787439/pulses/2537093687) by [Unito](https://www.unito.io)
mdibaiee commented 2 years ago

Also found this in Facebook Business SDK: https://github.com/facebook/facebook-python-business-sdk/blob/main/facebook_business/adobjects/adaccount.py#L3964