Extend Functionality to Extract Patient and Evaluators

[KTS-o7]

@krishna9304 This is just a rough plans on how to pass and get evals. Needs more work. Guide me from ur pov too please.

Like pydantic in python, we directly pass a Json Schema in typescript ig

Description

We need to extend the current functionality to extract patient and evaluator information in order to make good requests and get great responses from OpenAI. The current system and user prompts need to be updated to fit this purpose.

Current Functionality

The current functionality to send structured JSON schema requests to OpenAI is located in src/utils/openai.util.ts. Here is the relevant part of the code:

import axios from 'axios';
import { OpenAIModel } from 'src/station/schemas/patient.schema';

@Injectable()
export class OpenAiUtil {
  constructor(private readonly configService: ConfigService) {}

  url = 'https://api.openai.com/v1/chat/completions';
  headers = {
    'Content-Type': 'application/json',
    Authorization: `Bearer ${this.configService.get<string>('OPENAI_API_KEY')}`,
  };
  max_tokens = 200;
  temperature = 0.7;

  async getChatCompletion(
    prompt: Array<{ role: string; content: string }>,
    model: string,
  ) {
    const response = await axios.post(
      this.url,
      {
        max_tokens: this.max_tokens,
        temperature: this.temperature,
        messages: prompt,
        model,
      },
      { headers: this.headers },
    );
    const content: string = response.data.choices[0].message.content;
    return content;
  }

  async getChatCompletionsStream(
    prompt: Array<{ role: string; content: string }>,
    model: string,
  ) {
    const response = await axios.post(
      this.url,
      {
        messages: prompt,
        max_tokens: this.max_tokens,
        temperature: this.temperature,
        stream: true,
        model,
      },
      { headers: this.headers, responseType: 'stream' },
    );

    return response;
  }
}

Task Details

1. Create JSON Schema:

   • Define a JSON schema similar to the example provided:

     import { JSONSchemaType } from 'ajv';
     
     interface EvaluationResponse {
     steps: Array<{
      explanation: string;
      output: string;
     }>;
     final_answer: string;
     }
     
     const evaluationResponseSchema: JSONSchemaType<EvaluationResponse> = {
     type: 'object',
     properties: {
      steps: {
        type: 'array',
        items: {
          type: 'object',
          properties: {
            explanation: {
              type: 'string'
            },
            output: {
              type: 'string'
            }
          },
          required: ['explanation', 'output'],
          additionalProperties: false
        }
      },
      final_answer: {
        type: 'string'
      }
     },
     required: ['steps', 'final_answer'],
     additionalProperties: false
     };
     
     export default evaluationResponseSchema;

2. Update openai.util.ts:

   • Modify the methods to include the JSON schema:

     import axios from 'axios';
     import evaluationResponseSchema from './openai.schema';
     
     export async function sendRequest() {
     const requestBody = {
      model: "gpt-4o-2024-08-06",
      messages: [
        {
          role: "system",
          content: "You are a helpful evaluator assisting in a medical examination."
        },
        {
          role: "user",
          content: "Please provide an evaluation based on the following patient details."
        }
      ],
      response_format: {
        type: "json_schema",
        json_schema: evaluationResponseSchema
      }
     };
     
     try {
      const response = await axios.post('https://api.openai.com/v1/chat/completions', requestBody, {
        headers: {
          'Content-Type': 'application/json',
          Authorization: `Bearer YOUR_API_KEY`
        }
      });
      console.log(response.data);
     } catch (error) {
      console.error(error);
     }
     }

3. Extract Patient and Evaluator Data:

   • Retrieve patient and evaluator data from the database.
   • Format the data according to the schema provided for OpenAI requests.
   • Send the request using the updated openai.util.ts utility.

4. Example Code to Extract Data and Send Request:

   import { Patient } from 'src/station/schemas/patient.schema';
   import { Evaluator } from 'src/station/schemas/evaluator.schema';
   import { sendRequest } from 'src/utils/openai.util';
   
   async function getPatientData(patientId: string): Promise<Patient> {
    // Code to retrieve patient data from the database
    // Example: return await PatientModel.findById(patientId);
   }
   
   async function getEvaluatorData(evaluatorId: string): Promise<Evaluator> {
    // Code to retrieve evaluator data from the database
    // Example: return await EvaluatorModel.findById(evaluatorId);
   }
   
   async function formatAndSendRequest(patientId: string, evaluatorId: string) {
    const patient = await getPatientData(patientId);
    const evaluator = await getEvaluatorData(evaluatorId);
   
    const requestBody = {
      model: patient.openAiModel,
      messages: [
        {
          role: "system",
          content: `You are a helpful evaluator for station ${patient.associatedStation}.`
        },
        {
          role: "user",
          content: `Evaluate the patient with the following details: ${JSON.stringify(patient)}`
        }
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "evaluation_response",
          strict: true,
          schema: {
            type: "object",
            properties: {
              steps: {
                type: 'array',
                items: {
                  type: 'object',
                  properties: {
                    explanation: {
                      type: 'string'
                    },
                    output: {
                      type: 'string'
                    }
                  },
                  required: ["explanation", "output"],
                  additionalProperties: false
                }
              },
              final_answer: {
                type: 'string'
              }
            },
            required: ["steps", "final_answer"],
            additionalProperties: false
          }
        }
      }
    };
   
    await sendRequest(requestBody);
   }

References

• Patient Schema
• Evaluator Schema

---

[krishna9304]

FYI - the patient and the evaluator data is already fetched from the database here. If you plan to refactor the code for the openai utility then you need to make code changes at few places as it will affect the current flow.

Although the plan of getting structured output from openAi looks good to me. Please proceed with the above plan. Just a heads up before you make any changes - the output that it will generate is not consumable by the current implementation. You may need to change the schema for Evaluation and coordinate with @sufiyanas to adapt these changes on the client side.

[KTS-o7]

I noticed that this method needs too many changes. So for now I will first add a new structured output function to the openai file. Later on we need to see what and all types of outputs are needed

[krishna9304]

I noticed that this method needs too many changes. So for now I will first add a new structured output function to the openai file. Later on we need to see what and all types of outputs are needed

Sure, sounds good to me.