Investegate AI formatters

[DylanHalstead]

Look through and toy with the options for formatting AI to JSON and weigh pros and cons

[DylanHalstead]

3 main options:

• Guidance (Python)
• LangChain (Python)
• LandChain.JS
• Outlines

[DylanHalstead]

https://js.langchain.com/docs/expression_language/cookbook/prompt_llm_parser

[DylanHalstead]

Code using langchain.js:

import { OpenAI } from 'langchain/llms/openai';
import { PromptTemplate } from 'langchain/prompts';
import { StructuredOutputParser } from "langchain/output_parsers";
import { z } from "zod";
import { config } from 'dotenv';
config();

const model = new OpenAI({
  openAIApiKey: process.env.OPENAI_API_KEY,
  modelName: 'gpt-3.5-turbo-16k',
  maxTokens: -1,
});

const cardSchema = z.object({
  title: z.string().describe("briefly describes the story item"),
  description: z.string().describe("gives more detail about card if more context is needed"),
  story_points: z.number().int().positive().describe("points based on complexity which are in the Fibonacci sequence format (0, 1, 2, 3, 5, 8, 13). Most cards should be 1, 2, or 3 points."),
  assigned: z.array(z.string()).describe("leave this empty"),
});
const listSchema = z.object({
  title: z.string().describe("Sprint title with high-level overview"),
  cards: z.array(cardSchema),
});
const boardSchema = StructuredOutputParser.fromZodSchema(
  z.object({
    lists: z.array(listSchema),
  })
);
const formatInstructions = boardSchema.getFormatInstructions();

const templateString = `I'm making a project titled "{title}" with the description "{description}". Create me a "{detail_level}" storyboard with cards describing what needs to be done to complete the entire project from beggining to end: \n{format_instructions}`;
const boardTemplateConfig = {
  template: templateString,
  inputVariables: ["title", "description", "detail_level"],
};

const boardTemplate = new PromptTemplate({
  template: boardTemplateConfig.template
    ,
    inputVariables: boardTemplateConfig.inputVariables,
    partialVariables: {format_instructions: formatInstructions},
});

const chain = boardTemplate.pipe(model);
const result = await chain.invoke({
  title: 'E-commerce platform',
  description: 'A shopping platform with product listings, shopping cart, checkout process, payment gateway integration, order tracking, and user account management',
  detail_level: 'very detailed',
});
console.log(result);

Output:

```json
{
  "title": "E-commerce platform",
  "description": "A shopping platform with product listings, shopping cart, checkout process, payment gateway integration, order tracking, and user account management",       
  "lists": [
    {
      "title": "Sprint 1 - Setting up the basic structure",
      "cards": [
        {
          "title": "Create project folder structure",
          "description": "Set up the initial folder structure for the project",
          "story_points": 1,
          "assigned": []
        },
        {
          "title": "Initialize git repository",
          "description": "Set up a git repository to track changes",
          "story_points": 1,
          "assigned": []
        },
        {
          "title": "Install and configure package manager",
          "description": "Set up and configure a package manager (e.g., npm or yarn) for the project",
          "story_points": 1,
          "assigned": []
        }
      ]
    },
    {
      "title": "Sprint 2 - Implementing Product Listings",
      "cards": [
        {
          "title": "Create Product model",
          "description": "Define the attributes and methods for the Product model",
          "story_points": 2,
          "assigned": []
        },
        {
          "title": "Create Product listing page",
          "description": "Design and implement the page to display the list of products",
          "story_points": 3,
          "assigned": []
        },
        {
          "title": "Integrate product filtering",
          "description": "Implement the ability to filter products based on different criteria (e.g., category, price)",
          "story_points": 5,
          "assigned": []
        }
      ]
    },
    {
      "title": "Sprint 3 - Shopping Cart and Checkout Process",
      "cards": [
        {
          "title": "Create Shopping Cart model",
          "description": "Define the attributes and methods for the Shopping Cart model",
          "story_points": 2,
          "assigned": []
        },
        {
          "title": "Add 'Add to Cart' functionality",
          "description": "Implement the functionality to add products to the shopping cart",
          "story_points": 3,
          "assigned": []
        },
        {
          "title": "Design and implement the Checkout page",
          "description": "Create a page where users can review their cart and proceed to checkout",
          "story_points": 5,
          "assigned": []
        }
      ]
    },
    {
      "title": "Sprint 4 - Payment Gateway Integration",
      "cards": [
        {
          "title": "Research and select a payment gateway provider",
          "description": "Explore different payment gateway options and choose the most suitable one for integration",
          "story_points": 3,
          "assigned": []
        },
        {
          "title": "Integrate the selected payment gateway",
          "description": "Implement the necessary code to connect the payment gateway with the checkout process",
          "story_points": 8,
          "assigned": []
        },
        {
          "title": "Test the payment gateway integration",
          "description": "Perform various test transactions to ensure the payment gateway is working correctly",
          "story_points": 3,
          "assigned": []
        }
      ]
    },
    {
      "title": "Sprint 5 - Order Tracking and User Account Management",
      "cards": [
        {
          "title": "Implement order tracking functionality",
          "description": "Allow users to track the status of their orders",
          "story_points": 5,
          "assigned": []
        },
        {
          "title": "Design and implement user registration page",
          "description": "Create a page where users can register for an account",
          "story_points": 3,
          "assigned": []
        },
        {
          "title": "Create user profile page",
          "description": "Design and implement a page where users can view and edit their profile information",
          "story_points": 5,
          "assigned": []
        }
      ]
    }
  ]
}
```.

[DylanHalstead]

Okay i've read the docs for both and guidance seems to be better. LangChain lit all it does is alter your promt guidance is actually manipulating the prompt and acts as a middleware with your model (when local). It speeds up the AI, uses less tokens, while LangChain does the opposite.

[DylanHalstead]

Looked into another repo called outlines but it only works when the LLM is local, which would add too much tech debt. Interesting service if we were to make or run our own LLM though.

[DylanHalstead]

If the ai given is not a transformer model (> gpt-3) guidance cannot template. Downscaling from gpt-3.5-turbo-16k to text-davinci-003 and testing if reduced token amount still gets good prompt response. Worse comes to worse we add caching which feeds into another query

[DylanHalstead]

This was a lie, I need to run it locally.... Figuring out pytorch now

[DylanHalstead]

After doing some investigation, guidance seems to have a large learning curve and has kinks but it can guarantee JSON formatting. The system, I think, should work something like this:

• Start by asking for sprint names
• Plug those generated sprint names into a for loop to prompt for amount of cards based on the sprint
• plug into another for loop which will obtain a card in the correct format
• plug all the pieces back together to make the JSON

There are ways in guidance to give the model "memory" but idk I was thinking of just storing in variables, giving it a new prompt giving it said info, and calling it a day. Would probably be better to cache but not sure how difficult that is to set up

This, from a token level, is Very Expensive. Testing this cost $5 😵‍💫, so for now I think we can take the chance on it maybe working and just try catching, throwing error but def more testing should be done to see what can be done. Maybe figure out how to run a model ourselves and go from there

[DylanHalstead]

Another route we could take is maybe making a function gpt can route into? Don't think this is a great idea but something we may want to consider: https://openai.com/blog/function-calling-and-other-api-updates