PBI 2: Sorter Results and ChatGPT summaries - BACKEND

[ghost]

PBI 2: Sorter Results and ChatGPT Summaries - BACKEND

User Story: As Cathie, a project manager frequently overwhelmed with sorting through numerous emails daily, I want a system that automatically categorizes my emails and provides concise summaries for each category. This way, when I access my email dashboard, I can quickly grasp the essential content of each category without having to sift through every individual email, enabling me to focus on the most critical messages first and manage my time more efficiently.

Persona: Project managers or professionals who handle a high volume of daily emails and need a streamlined method to quickly identify and address the most crucial communications without getting bogged down by less important messages.

Feature: An automated system that categorizes incoming emails into distinct categories and provides a concise summary for each category, allowing users to quickly understand the key content of each group at a glance.

Business Value: Enhances productivity and time management for users by reducing the time and effort spent on email management. It allows users to prioritize their focus on emails that are critical to their work, thereby improving their efficiency and effectiveness in their roles.

- Collect categorized email outputs from the JARVIS-email-sorter, filter them to retain only the specified categories, and generate and store summaries using ChatGPT for each selected category so that the front end can quickly display this organized and summarized information to the user upon request.

Tasks:

Sorter Output Collection:

• [x] Implement a method to collect the output from the JARVIS-email-sorter. (Extant, but needs modification)
• [x] Parse the sorter's probability map output to extract the categorization of emails. (Extant, but needs modification)

Category Filtering:

• [x] Filter the extracted email categories to retain only 'Company Business/Strategy', 'Logistic Arrangements', 'Purely Personal', and 'Document editing/checking/collaboration'. (Trivial)
• [x] Organize emails into these four categories for further processing. (Trivial)

ChatGPT Integration and Summarization:

• [x] Develop a system to batch emails by category, considering the size limit of ChatGPT requests. (I'll write down what I end up doing, probably some form of truncation)
• [x] Create a function to send each category's emails to ChatGPT in a structured format, like JSON, for summarization. (Both of these require a bit of thought)
• [x] Retrieve summaries from ChatGPT and store them with the corresponding category. (Trivial)

Data Storage and Retrieval (Optional):

• [x] Ensure the backend can store and efficiently retrieve both the categorized emails and their summaries. (Could store on device, or send to Firebase if we're feeling spicy, Firebase is probably too much work.)

Performance and Error Handling:

• [x] Implement error handling and retries for interactions with both the sorter and ChatGPT to manage network issues or API limits. (Sorter already has logic, ChatGPT portion will likely use similar)

Testing and Optimization:

• [x] Perform load testing to ensure the system performs well under different volumes of emails.
• [x] Perform Integration testing to ensure full function of the pipeline.
• [x] Perform MOCK testing of ChatGPT responses for categories and associated JSONs.

Decision on Processing Sequence:

• [x] Decide based on performance testing whether to process sorting and ChatGPT summarization sequentially at each button press or to process sorting first and then do ChatGPT summarization on demand. (Need to see the delays we are experiencing before we pull the final trigger on the method for best user experience)

Emily Note: I will move to this task after completing my first PBI. If the first PBI ends up needing significant work, I'll get started on this one, but then finish it in Sprint 3.

Acceptance Criteria:

• [x] Sorter Integration: Successfully collect and parse sorter output, filtering for the four specified categories.
• [x] Summarization: Batch emails by category and generate summaries using ChatGPT, considering API limits.

These will be done via Mock and Integration Tests.

Timeline: By 03/08 (Friday):

• [x] Complete the collection and parsing of sorter output.

By 03/10 (Sunday):

• [x] Finalize the filtering logic to retain and organize emails into the specified categories.

By 03/13 (Wednesday):

• [x] Develop and test the batch processing and integration with ChatGPT for email summarization.

By 03/17 (Sunday):

• [x] Implement and test error handling for sorter and ChatGPT interactions.
• [x] Conduct initial load and integration testing to ensure system robustness.

By 03/20 (Wednesday):

• [x] Complete final testing, including MOCK tests for ChatGPT responses.
• [x] Ensure all acceptance criteria are met and prepare for a summary of the project's completion.

[ghost]

I have basically finished this, but I want to ensure understanding of file creation. right now all the files are created as "mocks", but I will need to make a dummy class to see how it performs on a real phone. Right now we use a 'localstorage' package. Ideally the actual files will be created, e.g: storageService.saveData(categoryKey, {'emails': categoryList}), will save a real file, that can be accessed. (In the test environment, the storage is mocked in memory to avoid creating garbage.)

This goes a bit beyond the requirements, but will be helpful since I finished majority of the code early. Further tests to handle different loads should also be implemented. I will talk further will luna about connecting these two halves.

[ghost]

Results of full throttle integration test on 100 emails.

Error generating summary for EmailCategory.companyBusinessStrategy: Exception: Rate limiting error. Request too large for gpt-3.5-turbo in organization org-gDzFzIDbqthmTj1mVn6HTTxX on tokens per min (TPM): Limit 160000, Requested 1346993. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more. Summary for EmailCategory.purelyPersonal: The emails appear to be related to a discussion about flight tickets, specifically inquiring about Air France and Continental flights to various destinations such as Amsterdam, Rome, and Warsaw. Additionally, there are personal messages expressing love and a request for a phone call. The sender's name appears to be Wincenty J Kaminski. No emails found for category: EmailCategory.personalButProfessional Error generating summary for EmailCategory.logisticArrangements: Exception: Failed to generate completion. Status code: 400 No emails found for category: EmailCategory.employmentArrangements Error generating summary for EmailCategory.documentEditingCheckingCollaboration: Exception: Failed to generate completion. Status code: 400 No emails found for category: EmailCategory.emptyMessageDueToMissingAttachment No emails found for category: EmailCategory.emptyMessage Summary for EmailCategory.purelyPersonal: The emails appear to be related to a discussion about flight tickets, specifically inquiring about Air France and Continental flights to various destinations such as Amsterdam, Rome, and Warsaw. Additionally, there are personal messages expressing love and a request for a phone call. The sender's name appears to be Wincenty J Kaminski.

Need to implement some more methods to handle a large drop.

[ghost]

The tests need to be optimized to handle a larger flow. 100 might be out of the scope of the project, but 20-30 should be manageable in a single batch.

[ghost]

It can now handle 100 emails, and technically infinite emails. That would be a bad idea to do however. This has been merged with luna's into [30-etl-emails-to-sorter](https://github.com/COSC481W-2024Winter/JARVIS/tree/30-etl-emails-to-sorter). They needed to be pre-merged before going to main because they had some interesting conflicts. I also added a screen to see the two processes working in the app on the phone. Luna's calls the sorter on the user's email, mine calls the sorter on the test emails and sends them to chatgpt.

[ghost]

Issue has been completed