Try to speak with chatgpt using json

[cypherkitty]

Aim: bring structure to a question, try to use json/yaml to describe hierarchically a complex question to an LLM. Expected result:

1. LLM has to understand the question on all levels
2. LLM has to use each level in json to get more context about the question
3. LLM probably has to provide reply in json with details describing reasoning of how each level used to get an answer.

Expected achievement:

• LLM has to use a question as an multistep instruction to find best possible answer
• the question in json can be improved by adding more details and levels (more context), which helps LLM understand question better (propmt engineering best practices will be very useful in this case)

Activity:

• some video
• this guy is endlessly talking about AI agents. We need to learn from him

Query format (let's imagine where we can start with json/yaml format for a query):

question:
  text: "how to write a hello world application on rust"
  context:
    - text: "I want to know what cargo is for"
       context:
         - text: "I want to know how to install rust and check that it works"
         - text: "I want to know how to create a new project with cargo"
            context:
              - I want to know what types of applications I can create with cargo
              - I want to know how to run my application
    - text: I wan't to know how to compile my application for different platforms
       context:
         - text: what platforms rust supports
         - text: how to install a new target via rustup

As you can see in the example above:

• on the root level we describe our question with little context, it's not clear how to interpret the question for an LLM.
• Then on next levels we describe more precisely what exactly we want LLM to do.
• The target is to get the response from LLM that takes into account all details we provided on all levels of the full context

[cypherkitty]

We need somehow force LLM to provide an answer with all details and LLM has to use whole question and can't give up on any parts of the query, and LLM has to have higher priority how higher levels (so, the most important question in our case is: "how to write a hello world application on rust". Other details helping make result better).

Would be great if LLM provides a report with details what part of query context used for particular parts of response

[cypherkitty]

https://youtu.be/T9aRN5JkmL8

[cypherkitty]

https://www.youtube.com/@Data-Centric/videos

[0xthecableguy]

I worked on this problem and here are my conclusions:

• the task sounds like “try to speak with chatgpt using json”
• the first question I solved was: what exactly should be in the json structure
• I got the following structure:

  
  #[derive(Serialize, Deserialize, Debug)]
  pub struct RequestStructure {
  pub request: String,
  pub cache: Vec<String>,
  pub context: String,
  pub viewpoints: Vec<String>,
  pub user_profile: UserProfile,
  }

[derive(Serialize, Deserialize, Debug)]

pub struct UserProfile { pub expertise_lvl: String, pub communication_style: String, }

, which I fill out _recursively_, starting with request (user request), which I simply add to the structure.

Next, I sequentially send the structure to the LLM with the tasks of filling out all the fields, in the end I get the following:

{ "request": "What do I need to write hello world in rust?", "cache": [ "[2024-09-27 23:26:24 (Friday)] User: What do I need to write hello world in rust?\nAssistant: " ], "context": "A user is interested in how to write a \"Hello, World!\" program in the Rust programming language. There is no conversation history, indicating that this is likely the user's first request on this topic. The user may be new to Rust and looking for basic information on how to get started programming in the language.", "viewpoints": [ "Necessary tools and development environment for writing a program in Rust", "Basic Rust syntax for writing a basic program", "The process of compiling and executing a program in Rust", "Installing and configuring the Rust and Cargo compiler", "Examples and resources for learning Rust for beginners" ], "user_profile": { "expertise_lvl": "basic", "communication_style": "conversational" } }

- the essence of the experiment: I tried to “enrich” the user’s request using the request itself, without adding any data from outside, otherwise I would have received the well-known RAG system
- I still plan to edit and test the meta-tasks that are sent to LLM to fill in the values ​​of the json structure, just like the structure fields themselves, but now for me they look the most adequate
- I’m still developing a universal chain of questions to compare the operation of the system with and without json, but here’s one example:

The question: 
"how to write hello world in rust?"

Answer using json structure:
https://chatgpt.com/share/66f6876b-5c8c-8011-a879-da811ff488d4

Answer without using j-son structure:
https://chatgpt.com/share/66f717b2-b164-8011-ad1f-412055b254ea