Implement Sequential Async Task Chain for Article Processing

[Devasy23]

Description:
This task involves building a complete asynchronous task pipeline for processing articles. Each article should be processed sequentially through multiple stages: fetching the content, summarizing, applying sentiment analysis, and extracting keywords. The processing should be independent for each article but should allow concurrent processing of multiple articles. Additionally, handle failure at any step by logging the error without stopping the whole pipeline.

Changes:

• Implement the overall pipeline function that chains the tasks for content fetching, summarization, sentiment analysis, and keyword extraction.
• Ensure that each article’s tasks are executed in sequence (content fetch → summarize → sentiment analysis → keyword extraction) but that multiple articles can be processed in parallel.
• Handle task failures at each stage without affecting the overall pipeline.

Example Dummy Code:

import asyncio
from aiohttp import ClientSession

async def process_article(article_id, url, session: ClientSession):
    try:
        content = await fetch_content(article_id, url, session)
        summary = await summarize_content(article_id, content)
        sentiment = await analyze_sentiment(article_id, summary)
        keywords = await extract_keywords(article_id, summary)

        print(f"Article {article_id} processed: Sentiment={sentiment}, Keywords={keywords}")
    except Exception as e:
        print(f"Failed to process article {article_id}: {e}")

# Function to process articles concurrently
async def process_articles_in_batches(article_urls):
    async with ClientSession() as session:
        tasks = [process_article(article['id'], article['url'], session) for article in article_urls]
        await asyncio.gather(*tasks)

# Usage example
async def example_usage():
    article_urls = [{'id': '123', 'url': 'http://example.com/article123'}, {'id': '124', 'url': 'http://example.com/article124'}]
    await process_articles_in_batches(article_urls)

[NikhilJ2005]

Can tou assign this issue to me @NikhilJ2005|Contributor GSSOC'24 Extended

[vrajpatelll]

Hi @NikhilJ2005, Thank you for your interest. This issue is connected to previous issues #39 and #40, so I recommend collaborating with @navsrijan, who is currently working on #39. Additionally, I suggest waiting for the upcoming issue regarding LLM crew-ai, as you have experience with that technology. We will be raising that issue very soon. Thank you.

[NikhilJ2005]

okay thanks