search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.72k stars 3.78k forks source link

[Source-netsuite] Missing date format for european dates #36821

Open sharingan-no-kakashi opened 2 months ago

sharingan-no-kakashi commented 2 months ago

Connector Name

netsuite

Connector Version

0.1.3

What step the error happened?

During the sync

Relevant information

Netsuite connector only supports "%m/%d/%Y" and "%Y-%m-%d" date formats

I think that adding in airbyte-integrations/connectors/source-netsuite/source_netsuite/constraints.py the following NETSUITE_INPUT_DATE_FORMATS: list[str] = ["%m/%d/%Y", "%Y-%m-%d", "%d/%m/%Y"] could solve the issue.

Relevant log output

> 2024-04-04 11:08:51 source > Stream `invoice`: cannot read or find the object. Skipping, full error message: Invalid search query. Detailed unprocessed description follows. Search error occurred: Parse of date/time "01/25/2020" failed with date format "d/M/yy" in time zone America/Los_Angeles
Caused by:
        java.text.ParseException: Unparseable date: "01/25/2020".
2024-04-04 11:08:52 source > Stream `invoice`: cannot read or find the object. Skipping, full error message: Invalid search query. Detailed unprocessed description follows. Search error occurred: Parse of date/time "02/15/2020" failed with date format "d/M/yy" in time zone America/Los_Angeles
Caused by: java.text.ParseException: Unparseable date: "02/15/2020".
2024-04-04 11:08:52 source > Stream `invoice`: cannot read or find the object. Skipping, full error message: Invalid search query. Detailed unprocessed description follows. Search error occurred: Parse of date/time "02/15/2020" failed with date format "d/M/yy" in time zone America/Los_Angeles

024-04-04 10:37:26 source > Stream `invoice`: cannot read using date format `%Y-%m-%d
2024-04-04 10:37:26 source > DATE FORMAT exception. Cannot read using known formats ['%m/%d/%Y', '%Y-%m-%d']
2024-04-04 10:37:26 source > Stream `invoice`: INVALID_PARAMETER error occured, full error message: Invalid search query. Detailed unprocessed description follows. Search error occurred: Parse of date/time "2020-02-01" failed with date format "d/M/yy" in time zone America/Los_Angeles
Caused by:
        java.text.ParseException: Unparseable date: "2020-02-01".
2024-04-04 10:37:26 source > Encountered an exception while reading stream invoice
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 287, in _read_full_refresh
    for _slice in slices:
  File "/airbyte/integration_code/source_netsuite/streams.py", line 280, in stream_slices
    slice_start = start.strftime(self.default_datetime_format)
  File "/airbyte/integration_code/source_netsuite/streams.py", line 54, in default_datetime_format
    return NETSUITE_INPUT_DATE_FORMATS[self.index_datetime_format]
IndexError: list index out of range
2024-04-04 10:37:26 source > Finished syncing invoice

Contribute

marcosmarxm commented 2 months ago

Thanks @sharingan-no-kakashi please chek my comments in your PR.