Implement Podcast Data Enrichment Pipeline Using OpenAI API

Description

This pull request introduces a comprehensive pipeline for enriching podcast episode data using the OpenAI API. Our primary goal is to extract and add insightful metadata to our existing podcast episode dataset, which includes fields like title, number, duration, and description. The enrichment involves adding information about episode participants, their genders, and general topics, obtained via AI-driven analysis.

Key Components

RSS File Parsing (rss_parser.py):

Parses podcast episode information from an RSS feed in XML format.
Extracts key details such as title, episode number, duration, and description.

Interacts with the OpenAI API to analyze podcast descriptions (api_integration.py)

Extracts structured information like participant names, genders, and episode topics.
Formats API requests and parses responses into a JSON structure for consistent processing.

Data Aggregation (data_aggregator.py) - Combines the parsed RSS data with API results.

Enriches the original dataset with new columns based on the API's response.
Offers functionality to process either the full dataset or a sampled subset.

Implementation Details

The pipeline is designed with a focus on modularity, allowing each component to function independently yet integrate seamlessly.
Special attention has been given to language processing, particularly for handling Hebrew in podcast descriptions.
The process concludes with exporting the enriched dataset, appending '_enriched' to the original filename, which provides a clear lineage of the data transformation.

Usage

The pipeline is primarily intended for internal use to enhance our podcast dataset. It can be triggered via the data_agregator.py script, which is configured to process a random sample of episodes for initial testing and can be easily adjusted for full-scale processing.

ivrit-ai / ivrit.ai