Closed amindadgar closed 1 month ago
The recent updates across multiple Python files enhance the ETL processes for Discord data extraction by incorporating detailed logging statements. These logs provide insights into the status and conditions of the data extraction, such as the availability of previous data and the extraction period, ensuring better traceability and debugging capabilities.
Files/Paths | Change Summary |
---|---|
dags/analyzer_helper/discord/discord_extract_raw_infos.py |
Added logging for data extraction conditions, imported logging , modified extract method to include prefix variable and various logging messages. |
dags/analyzer_helper/discord/discord_extract_raw_members.py |
Added logging for extraction process, imported logging . |
dags/discord_analyzer_etl.py |
Added logging statements for raw data and members extraction, adjusted logic for loading data into the database based on the presence of data. |
sequenceDiagram
participant Scheduler
participant DiscordAPI
participant Logger
participant Database
Scheduler->>DiscordAPI: Request data extraction
DiscordAPI->>Logger: Log status - checking previous data
alt Previous data exists
DiscordAPI->>Scheduler: Send extracted data
Scheduler->>Logger: Log data extraction completion
Scheduler->>Database: Load data if available
Database-->>Logger: Log data loading status
else No previous data
Logger-->>Scheduler: Log no data available
end
🎩 In the world of code, where data flows,
Logs now tell where the process goes.
Extracting infos and members with care,
Our ETL shines, now more aware.
Debugging's a breeze with logs in sight,
Celebrate the changes, everything's right. 🎩
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?
Summary by CodeRabbit
New Features
Bug Fixes