search

airbytehq / PyAirbyte

PyAirbyte brings the power of Airbyte to every Python developer.
https://docs.airbyte.com/pyairbyte
Other
235 stars 42 forks source link

[source-slack] IncrementalMessageStream for threads channel never stops syncing #430

Open whimo opened 1 month ago

whimo commented 1 month ago

Connector Name

source-slack

Connector Version

1.3.1

What step the error happened?

During the sync

Relevant information

Threads channel syncing never stops. It seems to cycle through all the channels, updating the end timestamp each loop (as some time passed while the data was queried), and does that over and over. Other channels work fine.

Using PyAirbyte:

import airbyte as ab

source = ab.get_source(
    "source-slack",
    config={
        "api_token": "<my-api-token>",
        "start_date": "2024-01-01T00:00:00Z",
        "lookback_window": 1,
        "join_channels": False,
    },
    install_if_missing=True,
)
source.check()
source.select_streams(["channels", "channel_members", "channel_messages", "threads"])

result = source.read()

for name, records in result.streams.items():
    print(f"Stream {name}: {len(list(records))} records")
    for record in records:
        print(record.get("text"))

Relevant log output

2024-10-22 15:40:57 - INFO - Check succeeded
2024-10-22 15:41:02 - INFO - Starting syncing SourceSlack
2024-10-22 15:41:02 - INFO - Marking stream channels as STARTED
2024-10-22 15:41:02 - INFO - Setting state of SourceSlack stream to {'__ab_no_cursor_state_message': True}
2024-10-22 15:41:02 - INFO - Syncing stream: channels 
2024-10-22 15:41:02 - INFO - Marking stream channels as RUNNING
2024-10-22 15:41:02 - INFO - Read 4 records from channels stream
2024-10-22 15:41:02 - INFO - Marking stream channels as STOPPED
2024-10-22 15:41:02 - INFO - Finished syncing channels
2024-10-22 15:41:02 - INFO - SourceSlack runtimes:
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:02 - INFO - Marking stream channel_members as STARTED
2024-10-22 15:41:02 - INFO - Setting state of SourceSlack stream to {'__ab_no_cursor_state_message': True}
2024-10-22 15:41:02 - INFO - Syncing stream: channel_members 
2024-10-22 15:41:02 - INFO - Marking stream channel_members as RUNNING
2024-10-22 15:41:03 - INFO - Read 18 records from channel_members stream
2024-10-22 15:41:03 - INFO - Marking stream channel_members as STOPPED
2024-10-22 15:41:03 - INFO - Finished syncing channel_members
2024-10-22 15:41:03 - INFO - SourceSlack runtimes:
Syncing stream channel_members 0:00:00.985201
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:03 - INFO - Marking stream channel_messages as STARTED
2024-10-22 15:41:03 - INFO - Setting state of SourceSlack stream to {'states': [{'partition': {'channel': 'C07Q9KX2U6T', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.094199}}, {'partition': {'channel': 'C07QHGM9B1C', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.118059}}, {'partition': {'channel': 'C07RCUS2MGQ', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.155319}}, {'partition': {'channel': 'C07RFRXUQN5', 'parent_slice': {}}, 'cursor': {'float_ts': 1729283939.911469}}]}
2024-10-22 15:41:03 - INFO - Syncing stream: channel_messages 
2024-10-22 15:41:03 - INFO - Marking stream channel_messages as RUNNING
2024-10-22 15:41:04 - INFO - Read 6 records from channel_messages stream
2024-10-22 15:41:04 - INFO - Marking stream channel_messages as STOPPED
2024-10-22 15:41:04 - INFO - Finished syncing channel_messages
2024-10-22 15:41:04 - INFO - SourceSlack runtimes:
Syncing stream channel_members 0:00:00.985201
Syncing stream channel_messages 0:00:01.123485
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:04 - INFO - Marking stream threads as STARTED
2024-10-22 15:41:04 - INFO - Setting state of SourceSlack stream to {}
2024-10-22 15:41:04 - INFO - Syncing stream: threads 
2024-10-22 15:41:04 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:04 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:04 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:05 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600864.626089}
2024-10-22 15:41:05 - INFO - Marking stream threads as RUNNING
2024-10-22 15:41:07 - INFO - Reading the channel: `social`
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600867.384261}
2024-10-22 15:41:08 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:08 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:09 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:09 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600868.85753}
2024-10-22 15:41:10 - INFO - Reading the channel: `support`
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600870.328075}
2024-10-22 15:41:12 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600872.45988}
2024-10-22 15:41:14 - INFO - Reading the channel: `social`
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600874.548036}
2024-10-22 15:41:15 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600875.567569}
2024-10-22 15:41:16 - INFO - Reading the channel: `support`
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600876.609494}
2024-10-22 15:41:18 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600878.290911}
2024-10-22 15:41:20 - INFO - Reading the channel: `social`
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600880.449153}
2024-10-22 15:41:21 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600881.485752}
2024-10-22 15:41:22 - INFO - Reading the channel: `support`
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600882.548746}
2024-10-22 15:41:24 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600884.416193}
2024-10-22 15:41:27 - INFO - Reading the channel: `social`
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600887.079985}

Contribute