As a Developer, I want to be able to perform end-to-end tests of our processing pipeline.

[Endlessflow]

Issue: As a Developer, I want to be able to perform end-to-end tests of our processing pipeline.

Description:

To ensure the reliability and performance of our image-to-form processing pipeline, we are working on a robust testing framework. As part of our testing framework, we want a way to run end-to-end tests that encompass various components of the pipeline, such as OCR and LLM models. The primary goal is to facilitate seamless testing when changes are made to different engines, models, or prompts.

Objectives:

1. Create a Script for Automated Testing:

   • Develop a script that automatically checks the sample folder inside input_images and verifies if the existing test outputs are available for each image.
   • If test outputs are not found, the script will execute the end-to-end test for that image.

2. Organize Sample Folder Structure:

   • Structure the sample folder as follows:

     sample\
     |___ input_images\
     |     |___ img1.png
     |     |___ ...
     |___ ocr_outputs\
     |     |___ document_intelligence\
     |          |___ ocr_document_intelligence_img1.txt
     |          |___ ...
     |___ llm_outputs\
     |     |___ gpt3_5\
     |          |___ llm_gpt3_img1.txt
     |          |___ ...

3. Focus on LLM Model Testing:

   • Initially prioritize testing different LLM models (e.g., GPT-3.5) to address the urgent need for accessing and validating various LLM outputs.

4. Future Expansion:

   • Plan for future enhancements may include testing with different OCR models and prompts, although this is not a priority for the immediate implementation.

Acceptance Criteria:

• A script that can identify the need for running tests based on the presence or absence of existing outputs.
• Organized folder structure for storing input images and their corresponding OCR and LLM outputs.
• Ability to run end-to-end tests for different LLM models.