1.1.1 Write Data Extraction Loop

[earth2travis]

Develop a loop that iterates through the relevant endpoints of the Boardroom Governance API to extract data. The loop should handle API requests, process responses, and store the extracted data in a structured format. This automated data extraction loop is crucial for ensuring efficient and continuous data retrieval from the API.

[santteegt]

Currently exploring Airbyte and its low-code custom connectors. This integration will allow us to support different types of API pagination techniques seamlessly and reduce dev time on implementing different API data extraction scenarios

[earth2travis]

• Still working on this
• Decided the end product should be able to extract from any API
  • Note: the client requested this feature
• Need to consider different auth and pagination techniques
• Decided to switch to the Airbyte Apache Doris ELT data integration destination connection
• There local connectors feature allows you to define a YAML file for the endpoints and handles everything under the hood
• These will be the entry points for the pipeline
• Will replace the Pathway HTTP connector
• Pathway is still core for other features related to transforming, loading, and generating embeddings
• Need to run some tests after plugging in data extraction loop

[santteegt]

• Finished implementing a custom connector using PyAirbyte, declarative HttpRequester and Pathway custom Python connectors.
• Tested with proposals and discourseTopicPosts
• Integrated into the pipeline