airbytehq / PyAirbyte

PyAirbyte brings the power of Airbyte to every Python developer.
https://docs.airbyte.com/pyairbyte
Other
216 stars 31 forks source link

🐛 Bug: Unexpected failure when `select_streams()` is called before `set_config()` #51

Closed aaronsteers closed 2 months ago

aaronsteers commented 7 months ago

Ideally, these should be runnable in either order.

Traceback log:

Traceback (most recent call last):
  File "/Users/ajsteers/Source/aj-dataops-personal/get_data.py", line 35, in <module>
    main()
  File "/Users/ajsteers/Source/aj-dataops-personal/get_data.py", line 31, in main
    read_result = get_github_data(cache=cache)
  File "/Users/ajsteers/Source/aj-dataops-personal/get_data.py", line 17, in get_github_data
    source_github.select_streams(["pull_requests", "issues"])
  File "/Users/ajsteers/Source/aj-dataops-personal/.venv/lib/python3.10/site-packages/airbyte/source.py", line 123, in select_streams
    available_streams = self.get_available_streams()
  File "/Users/ajsteers/Source/aj-dataops-personal/.venv/lib/python3.10/site-packages/airbyte/source.py", line 198, in get_available_streams
    return [s.name for s in self.discovered_catalog.streams]
  File "/Users/ajsteers/Source/aj-dataops-personal/.venv/lib/python3.10/site-packages/airbyte/source.py", line 251, in discovered_catalog
    self._discovered_catalog = self._discover()
  File "/Users/ajsteers/Source/aj-dataops-personal/.venv/lib/python3.10/site-packages/airbyte/source.py", line 179, in _discover
    with as_temp_files([self._config]) as [config_file]:
  File "/Users/ajsteers/Source/aj-dataops-personal/.venv/lib/python3.10/site-packages/airbyte/source.py", line 165, in _config
    raise exc.AirbyteConnectorConfigurationMissingError(
airbyte.exceptions.AirbyteConnectorConfigurationMissingError: AirbyteConnectorConfigurationMissingError: Connector is missing configuration.

It is true that for many sources we cannot validate that the requested streams exist until after valid credentials are given. However, a better solution might be to print a warning if stream names can't be validated, rather than raising a hard failure.

Suraj-Vishwakarma70 commented 3 months ago

Hey @aaronsteers Please assign it to me!!

Suraj-Vishwakarma70 commented 2 months ago

Thanks for all the support @aaronsteers !