Closed lewisdawson closed 2 years ago
The issue may be with with the print()
statement in entrypoint.py:
def launch(source: Source, args: List[str]):
source_entrypoint = AirbyteEntrypoint(source)
parsed_args = source_entrypoint.parse_args(args)
for message in source_entrypoint.run(parsed_args):
print(message)
It prints every single row consumed. Perhaps there could be a flag for this or this could be a debug logging statement?
@lewisdawson would you be able to upgrade Airbyte to the newest version to see if the issue persists then?
Did a bit more digging into this and figured out the issue. Was related to how my source connector was written. In short, when the HttpStream.parse_response()
function uses return
statements to return data, all the records get printed to stdout. When I modified the function to use yield
statements to return data, each record is no longer printed.
This issue is good to close out, however, if anyone with more knowledge of Airbyte understands why this happens, would love to get an explanation on this behavior before this is closed out.
I'm glad you were able to figure it out, and thank you for the detailed writeup! I will close the issue as it's been solved, but feel free to post on the forum or in Slack to continue this discussion!