search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.4k stars 3.97k forks source link

[source-adjust] time date format is wrong after migration to lowcode #41054

Open tim-werner opened 1 month ago

tim-werner commented 1 month ago

Connector Name

source-adjust

Connector Version

0.1.4, 0.1.5,0.1.6

What step the error happened?

Configuring a new connector

Relevant information

The time date format that is passed to the adjust API is wrong. The adjust connector passes a date '%Y-%m-%d' to the API but the API expects the format like '%Y-%m-%dT%H:%M:%SZ'.

The fix would be possible to change the manifest.yaml for the DatetimeBasedCursor from datetime_format: "%Y-%m-%d" to datetime_format: "%Y-%m-%dT%H:%M:%SZ".

Relevant log output

Encountered an error trying to connect to stream AdjustReport. Error:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/checks/check_stream.py", line 42, in check_connection
stream_is_available, reason = availability_strategy.check_availability(stream, logger, source)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 36, in check_availability
stream_slice = get_first_stream_slice(stream)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/utils/stream_helper.py", line 21, in get_first_stream_slice
stream.stream_slices(
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 148, in stream_slices
return self.retriever.stream_slices()
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 378, in stream_slices
return self.stream_slicer.stream_slices()
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py", line 168, in stream_slices
start_datetime = self._calculate_earliest_possible_value(self._select_best_end_datetime())
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py", line 173, in _calculate_earliest_possible_value
earliest_possible_start_datetime = min(self._start_datetime.get_datetime(self.config), end_datetime)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/datetime/min_max_datetime.py", line 58, in get_datetime
time = self._parser.parse(str(self.datetime.eval(config, **additional_parameters)), datetime_format) # type: ignore # datetime is always cast to an interpolated string
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/datetime/datetime_parser.py", line 33, in parse
parsed_datetime = datetime.datetime.strptime(str(date), format)
File "/usr/local/lib/python3.9/_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File "/usr/local/lib/python3.9/_strptime.py", line 349, in _strptime
raise ValueError("time data %r does not match format %r" %
ValueError: time data '2024-07-01' does not match format '%Y-%m-%dT%H:%M:%SZ'

Contribute

marcosmarxm commented 1 month ago

Thanks for the quick fix @tim-werner