New Source Connector: Jina "Reader API" for web scraping and feeding LLM models

aaronsteers commented 3 months ago

Overview

The most popular web scraping tool source connector right now is Apify. However, this new API from Jina is focused specifically on LLM use cases and it helpfully outputs markdown which is easy for humans and LLMs to work with. It also doesn't (yet) require a paid account.

The goal is to create a connector which could be used by Airbyte users to leverage this API.

Technical spec

You would write a new source connector which can connect to API and get the scraped content, allowing Airbyte users to send this data downstream to any Airbyte destination.

Notes:

One point of ambiguity is whether this source would be able to scrape links recursively or hierarchically, or if it would need the user to provide discreet URLs for each source that is intended to be scraped.
- Whoever does pick this up, please let us know during development which direction you go on this design choice - and/or let us know if you have any questions.

Definition of Done

You would build a new "Jina Reader API" source in Python which can scrape text from websites.
The streams' primary keys should be the source URL that was scraped.
Incremental processing is probably not possible - so incremental keys would not be required. (Syncs should always be in 'full table' mode.)
You should use the Python CDK as much as possible.
The connector should pass integration tests and acceptance tests.

btkcodedev commented 3 months ago

I would like to take this if possible, this is a rest API type, right? is the response dynamic?

btkcodedev commented 3 months ago

Yes it has a REST API interface and JSON output, Ref: https://jina.ai/reader/#apiform I would like to take this issue CC: @marcosmarxm, there is no previous assignee, I won't miss this time :man_dancing:

aaronsteers commented 3 months ago

@btkcodedev - It's yours! Thanks for jumping in. I'm excited about this one for sure! Let @marcosmarxm or I know if you have any questions along the way!

btkcodedev commented 3 months ago

Linking PR: https://github.com/airbytehq/airbyte/pull/39515

btkcodedev commented 3 months ago

CC: @marcosmarxm @bindipankhudi @aaronsteers :bow: Thanks!!

bindipankhudi commented 3 months ago

Thank you @btkcodedev! Assigning to @aaronsteers for review.

aaronsteers commented 3 months ago

@btkcodedev - This is looking awesome! I added a few comments + suggestions to the PR. 🚀

airbytehq / PyAirbyte-Hackathon