parser for niche site - Githubissues

I am a helpful assistant! How can I help?

write me something that will grab chunks of a html in increments of a token size limit until it iterates through the entire html

I understand that you want to process an HTML string in chunks based on a token size limit. Here's a Python function that does that:

def process_html_in_chunks(html, token_size_limit):
    start = 0
    end = token_size_limit
    html_length = len(html)

    while start < html_length:
        chunk = html[start:end]
        # Process the chunk here
        print(chunk)

        start += token_size_limit
        end += token_size_limit

html = "<html><head><title>Example</title></head><body><p>Hello, World!</p></body></html>"
token_size_limit = 10
process_html_in_chunks(html, token_size_limit)

This function takes an HTML string and a token size limit as input. It iterates through the HTML...

Do it in typescript

Sure, here's the same functionality implemented in TypeScript:

function processHtmlInChunks(html: string, tokenSizeLimit: number): void {
  let start = 0;
  let end = tokenSizeLimit;
  const htmlLength = html.length;

  while (start < htmlLength) {
    const chunk = html.slice(start, end);
    // Process the chunk here
    console.log(chunk);

    start += tokenSizeLimit;
    end += tokenSizeLimit;
  }
}

const html = "<html><head><title>Example</title></head><body><p>Hello, World!</p></body></html>";
const tokenSizeLimit = 10;
processHtmlInChunks(html, tokenSizeLimit);

This TypeScript function processHtmlInChunks takes an HTML string and a token size limit as input. It iterates through the HTML string...

No feed each chunk into chatgpt

To feed each chunk into ChatGPT, you'll need to make API calls to OpenAI's GPT-3. Here's an example using TypeScript with the node-fetch library to make the API calls. First, install the node-fetch library:

...

---

instead of that use this as an example

import { AxiosError } from "axios"; import { getOpenAI } from "lib/openai"; import { ChatCompletionRequestMessage, ChatCompletionRequestMessageRoleEnum, CreateChatCompletionRequest } from "openai";

export const NAME_MAX_TOKEN_LENGTH = 10;

export const hasCodeBlock = (llmOutput: string) => { // check for a code block // if so return true return llmOutput.includes(""); };

export const promptBatching = (...args: string[]) => { return args.join("\n\n"); };

export const promptBatchingWithNumbering = (...args: string[]) => { return args.map((arg, index) => ${index + 1}. ${arg}).join("\n\n"); };

// TODO(@steven4354): getFunctionNameRegex doesn't work on retrieveNewLogs. why? export const getFunctionNameRegex = (inputString: string): string | null => { const functionNameRegex = /^[a-zA-Z$][0-9a-zA-Z$]*\b/; const match = inputString.match(functionNameRegex);

return match ? match[0] : null; };

export const simpleSingleQueryGPTCall = async (prompt: string, options?: Partial) => { const openai = getOpenAI();

let history: Array = [ { role: ChatCompletionRequestMessageRoleEnum.User, content: prompt, }, ]; let completionParams: CreateChatCompletionRequest = { model: "gpt-3.5-turbo", messages: history, ...options, }; const completion = await openai.createChatCompletion(completionParams);

return completion.data.choices[0].message?.content || ""; };

// takes history and makes it 1 string line export const convertHistoryToLog = (history: Array) => { return history.map(({ role, content }) => ${role}: ${content}).join("\n\n"); };

export const simpleChatGPTCall = async ( history: Array, options?: Partial, retries: number = 5, delay: number = 500, ) => { const openai = getOpenAI();

let completionParams: CreateChatCompletionRequest = { model: "gpt-3.5-turbo", messages: history, temperature: 0, max_tokens: 800, top_p: 1, frequency_penalty: 0, presence_penalty: 0, ...options, };

for (let i = 0; i < retries; i++) { try { const completion = await openai.createChatCompletion(completionParams); return completion.data.choices[0].message?.content || ""; } catch (error) { if (isAxiosError(error) && error?.response?.status === 429 && i < retries - 1) { await sleep(delay); delay *= 2; // Exponential backoff } else { throw error; } } }

throw new Error('Max retries reached'); };

export const sleep = (ms: number) => { return new Promise((resolve) => setTimeout(resolve, ms)); };

export const isAxiosError = (error: unknown): error is AxiosError => { return (error as AxiosError).isAxiosError !== undefined; };

export const wrapInTypescript = (code: string) => { return "typescript" + "\n" + code + "\n" + "" + "\n"; };

export const wrapInCodeBlock = (code: string, language: string = "") => { return "" + language + "\n" + code + "\n" + "" + "\n"; };


now, rewrite your html chunk code

---

I see that you're using the OpenAI API to interact with GPT-3. Here's a modified version of the `processHtmlInChunks` function that sends each chunk to GPT-3 using the `simpleSingleQueryGPTCall` function from your example:

```typescript
import { simpleSingleQueryGPTCall } from './your_example_module';

async function processHtmlInChunks(html: string, tokenSizeLimit: number): Promise<void> {
  let start = 0;
  let end = tokenSizeLimit;
  const htmlLength = html.length;

  while (start < htmlLength) {
    const chunk = html.slice(start, end);

    try {
      const gptResponse = await simpleSingleQueryGPTCall(chunk);
      console.log('GPT-3 response:', gptResponse);
    } catch (error) {
      console.error('Error calling GPT-3:', error);
    }

    start += tokenSizeLimit;
    end += tokenSizeLimit;
  }
}

const html = "<html><head><title>Example</title></head><body><p>Hello, World!</p></body></html>";
const tokenSizeLimit = 10;
processHtmlInChunks(html, tokenSizeLimit);

Make sure to replace './your_example_module' with the correct path to the module containing the simpleSingleQueryGPTCall function and other utility functions.

This TypeScript function processHtmlInChunks takes an HTML string and a token size limit as input. It iterates through the HTML string in chunks and sends each chunk to GPT-3 using the simpleSingleQueryGPTCall function. The GPT-3 response is then logged to the console.

ok now what i want it to do is iterate through each chunk

feed it into gpt3

and at the end I should get a list of instructions in this format

click <id>
type <string>
type enter

To achieve this, you can modify the processHtmlInChunks function to accumulate the GPT-3 responses in an array and then process the responses to generate the desired list of instructions. Here's the updated function:

import { simpleSingleQueryGPTCall } from './your_example_module';

async function processHtmlInChunks(html: string, tokenSizeLimit: number): Promise<void> {
  let start = 0;
  let end = tokenSizeLimit;
  const htmlLength = html.length;
  const gptResponses: string[] = [];

  while (start < htmlLength) {
    const chunk = html.slice(start, end);

    try {
      const gptResponse = await simpleSingleQueryGPTCall(chunk);
      gptResponses.push(gptResponse);
    } catch (error...

---

you need the prompt as well what would the prompt tell gpt3 to do?

---

To instruct GPT-3 to generate a list of instructions based on the HTML chunks, you can modify the prompt to include a clear task description. Here's an example of how you can update the `processHtmlInChunks` function with a prompt:

```typescript
import { simpleSingleQueryGPTCall } from './your_example_module';

async function processHtmlInChunks(html: string, tokenSizeLimit: number): Promise<void> {
  let start = 0;
  let end = tokenSizeLimit;
  const htmlLength = html.length;
  const gptResponses: string[] = [];

  while (start < htmlLength) {
    const chunk = html.slice(start, end);

    const prompt = `Analyze the following HTML chunk and provide a list of instructions to interact with the elements in the format "click <id>", "type <string>", or "type enter":\n\n${chunk}`;

    try {
      const gptResponse = await simpleSingleQueryGPTCall(prompt);
      gptResponses.push(gptResponse);
    } catch (error) {
      console.error('Error calling GPT-3...

---

The prompt needs to be updated. you aren't telling it to randomly interact with the page, you want gpt3 to give instructions for a goal

example goal could be "Put in the address search bar this address 4949 san juan avenue"

also how do you aggregate it? what if some chunks have no html that would help with the task? what if you have multiple chunks that seem to conflict with instructions?

---

You're right, the prompt should be more specific about the goal. To address the issues you mentioned, you can update the prompt to include the goal and ask GPT-3 to consider the entire HTML content while providing non-conflicting instructions. You can also filter out empty or irrelevant responses before aggregating the results.

Here's an updated version of the `processHtmlInChunks` function:

```typescript
import { simpleSingleQueryGPTCall } from './your_example_module';

async function processHtmlInChunks(html: string, tokenSizeLimit: number): Promise<void> {
  let start = 0;
  let end = tokenSizeLimit;
  const htmlLength = html.length;
  const gptResponses: string[] = [];
  const goal = 'Put in the address search bar this address 4949 San Juan Avenue';

  while (start < htmlLength) {
    const chunk = html.slice(start, end);

    const prompt = `Given the goal "${goal}", analyze the following HTML chunk and provide a list of instructions to interact with the elements in the format "click <id>", "type <string>", or "type enter". Consider the entire HTML content and ensure the instructions do not conflict with each other:\n\n${chunk}`;

    try {
      const gptResponse = await simpleSingleQueryGPTCall(prompt);
      if (gptResponse.trim() !== '') {
        gptResponses.push(gptResponse);
      }
   ...

---

You forgot that gpt3 has a token size limit so it cant take in the entire html page, you need a different solution to sovle the problem.

You may talk about multiple approaches, then pick the best one to go with.

For each approach list out what the approach entails briefly without code

Then for the approach you choose, then code that one out so you only code once but analyze multiple approaches

---

You're right, GPT-3 has a token size limit, and we need to consider that while designing the solution. Here are three different approaches to handle this situation:

1. **Iterative approach**: Divide the HTML into chunks and send each chunk to GPT-3 one by one. After receiving the response for each chunk, update the instructions based on the new information. This approach might not be efficient, as it requires multiple API calls and may result in conflicting instructions.

2. **Summarize HTML**: Summarize the HTML content to fit within GPT-3's token limit and send the summarized version to GPT-3. This approach might miss some important details from the original HTML, but it can provide a more coherent set of instructions.

3. **Selective chunking**: Identify the most relevant parts of the HTML (e.g., form elements, buttons, and input fields) and create smaller chunks containing only those parts. Send these smaller chunks to GPT-3 to generate instructions. This approach can provide more accurate instructions while staying within the token limit.

I recommend going with the **Selective chunking** approach, as it focuses on the most relevant parts of the HTML and can provide more accurate instructions without exceeding the token limit.

Here's the updated `processHtmlInChunks` function using the selective chunking approach:

```typescript
import { simpleSingleQueryGPTCall } from './your_example_module';

function extractRelevantHtml(html: string): string[] {
  const parser = new DOMParser();
  const doc = parser.parseFromString(html, 'text/html');
  const relevantElements = doc.querySelectorAll('input, button, form');
  const relevantHtml: string[] = [];

  relevantElements.forEach((element) => {
    relevantHtml.push(element.outerHTML);
  });

  return relevantHtml;
}

async function processHtmlInChunks(html: string): Promise<void> {
  const relevantHtml = extractRelevantHtml(html);
  const gptResponses: string[] = [];
  const goal = 'Put in the address search bar this address 4949 San Juan Avenue';

  for (const chunk of relevantHtml) {
    const prompt = `Given the goal "${...

---

please start from the beginngin of the code and finish

---

Here's the complete code for the selective chunking approach, starting with the necessary imports and utility functions, followed by the `extractRelevantHtml` function and the updated `processHtmlInChunks` function:

```typescript
import { simpleSingleQueryGPTCall } from './your_example_module';

function extractRelevantHtml(html: string): string[] {
  const parser = new DOMParser();
  const doc = parser.parseFromString(html, 'text/html');
  const relevantElements = doc.querySelectorAll('input, button, form');
  const relevantHtml: string[] = [];

  relevantElements.forEach((element) => {
    relevantHtml.push(element.outerHTML);
  });

  return relevantHtml;
}

async function processHtmlInChunks(html: string): Promise<void> {
  const relevantHtml = extractRelevantHtml(html);
  const gptResponses: string[] = [];
  const goal = 'Put in the address search bar this address 4949 San Juan Avenue';

  for (const chunk of relevantHtml) {
    const prompt = `Given the goal "${goal}", analyze the following HTML chunk and provide a list of instructions to interact with the elements in the format "click <id>", "type <string>", or "type enter". Ensure the instructions do not conflict with each other:\n\n${chunk}`;

    try {
      const gptResponse = await simpleSingleQueryGPTCall(prompt);
      if (gptResponse.trim() !== '') {
        gptResponses.push(gptResponse);
      }
    } catch (error) {
      console.error('Error calling GPT-3:', error);
    }
  }

  // Aggregate and process the GPT-3 responses
  const instructions = gptResponses.join('\n');
  console.log('Instructions:', instructions);
}

const html = "<html><head><title>Example</title></head><body><form><input type='text' id='address-search' placeholder='Enter address'><button type='submit' id='search-btn'>Search</button></form></body></html>";
processHtmlInChunks(html);

This code first extracts the relevant parts of the HTML using the extractRelevantHtml function. Then, it sends each relevant chunk to GPT-3 using the simpleSingleQueryGPTCall function with an updated prompt that includes the goal. Finally, it aggregates and processes the GPT-3 responses to generate a list of instructions.

One thing, different goals may have different relevant elements. Maybe have gpt3 figure out the relevant elements based on goal?

for example, if the goal was "This page has info on the median income in this neighborhood, give me that information" then the relevant elements might be different

You're right, different goals may require different relevant elements. To address this, you can modify the prompt to ask GPT-3 to identify the relevant elements based on the goal. Then, you can extract those elements from the HTML and send them to GPT-3 for further instructions.

Here's an updated version of the processHtmlInChunks function that incorporates this change:

async function identifyRelevantElements(html: string, goal: string): Promise<string[]> {
  const prompt = `Given the goal "${goal}", identify the relevant HTML elements (e.g., input, button, form, div, span) that should be considered for generating instructions to achieve the goal. The HTML content is as follows:\n\n${html}`;
  const gptResponse = await simpleSingleQueryGPTCall(prompt);

  // Extract the relevant elements from the GPT-3 response
  const relevantElements = gptResponse.split(',').map((element) => element.trim());
  return relevantElements;
}

async function extractRelevantHtml(html: string, relevantElements: string[]): Promise<string[]> {
  const parser = new DOMParser();
  const doc = parser.parseFromString(html, 'text/html');
  const relevantHtml: string[] = [];

  relevantElements.forEach((element) => {
    const elements = doc.querySelectorAll(element);
    elements.forEach((el) => {
      relevantHtml.push(el.outerHTML);
    });
  });

  return relevantHtml;
}

async function processHtmlInChunks(html: string, goal: string): Promise<void> {
  const relevantElements = await identifyRelevantElements(html, goal);
  const relevantHtml = await extractRelevantHtml(html, relevantElements);
  const gptResponses: string[] = [];

  for (const chunk of relevantHtml) {
    const prompt = `Given the goal "${goal}", analyze the following HTML chunk and provide a list of instructions to interact with the elements in the format "click <id>", "type <string>", or "type enter". Ensure the instructions do not conflict with each other:\n\n${chunk}`;

    try {
      const gptResponse = await simpleSingleQueryGPTCall(prompt);
      if (gptResponse.trim() !== '') {
        gptResponses.push(gptResponse);
      }
    } catch (error) {
      console.error('Error calling GPT-3:', error);
    }
  }

  // Aggregate and process the GPT-3 responses
  const instructions = gptResponses.join('\n');
  console.log('Instructions:', instructions);
}

const html = "<html><head><title>Example</title></head><body><form><input type='text' id='address-search' placeholder='Enter address'><button type='submit' id='search-btn'>Search</button></form><div id='median-income'>The median income in this neighborhood is $75,000.</div></body></html>";
const goal = 'This page has info on the median income in this neighborhood, give me that information';
processHtmlInChunks(html, goal);

In this updated version, the identifyRelevantElements function sends the entire HTML content and the goal to GPT-3, asking it to identify the relevant elements. Then, the extractRelevantHtml function extracts the relevant elements from the HTML based on GPT-3's response. Finally, the processHtmlInChunks function sends the relevant HTML chunks to GPT-3 for generating instructions.

if you were to make the instructions how would you make it better? imagine each instruction ties to a actual pupeteer functionality.

write the code for this

To improve the instructions and make them more suitable for use with Puppeteer, you can modify the GPT-3 prompt to generate instructions in a more structured format, such as JSON. This will make it easier to parse the instructions and map them to Puppeteer functions.

Here's an updated version of the processHtmlInChunks function that generates JSON-formatted instructions:

async function processHtmlInChunks(html: string, goal: string): Promise<void> {
  const relevantElements = await identifyRelevantElements(html, goal);
  const relevantHtml = await extractRelevantHtml(html, relevantElements);
  const gptResponses: string[] = [];

  for (const chunk of relevantHtml) {
    const prompt = `Given the goal "${goal}", analyze the following HTML chunk and provide a list of instructions to interact with the elements in a JSON format. Each instruction should have an "action" (e.g., "click", "type", "typeEnter") and a "selector" (e.g., "#elementId", ".elementClass"). Ensure the instructions do not conflict with each other:\n\n${chunk}`;

    try {
      const gptResponse = await simpleSingleQueryGPTCall(prompt);
      if (gptResponse.trim() !== '') {
        gptResponses.push(gptResponse);
      }
    } catch (error) {
      console.error('Error calling GPT-3:', error);
    }
  }

  // Aggregate and process the GPT-3 responses
  const instructions = gptResponses.map((response) => JSON.parse(response));
  console.log('Instructions:', instructions);

  // Execute the instructions using Puppeteer
  await executeInstructions(instructions);
}

async function executeInstructions(instructions: any[]): Promise<void> {
  const puppeteer = require('puppeteer');
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  for (const instruction of instructions) {
    const action = instruction.action;
    const selector = instruction.selector;

    switch (action) {
      case 'click':
        await page.click(selector);
        break;
      case 'type':
        const text = instruction.text;
        await page.type(selector, text);
        break;
      case 'typeEnter':
        await page.type(selector, String.fromCharCode(13));
        break;
      default:
        console.error('Unknown action:', action);
    }
  }

  // Close the browser after executing the instructions
  await browser.close();
}

const html = "<html><head><title>Example</title></head><body><form><input type='text' id='address-search' placeholder='Enter address'><button type='submit' id='search-btn'>Search</button></form><div id='median-income'>The median income in this neighborhood is $75,000.</div></body></html>";
const goal = 'This page has info on the median income in this neighborhood, give me that information';
processHtmlInChunks(html, goal);

In this updated version, the GPT-3 prompt asks for JSON...

extend the amount of functionality, I also want the ability to extract text from a html tag etc and return it as the output

To extend the functionality and allow GPT-3 to extract text from an HTML tag, you can update the prompt to include this additional action. Here's an updated version of the processHtmlInChunks function that supports extracting text from an HTML tag:

async function processHtmlInChunks(html: string, goal: string): Promise<void> {
  const relevantElements = await identifyRelevantElements(html, goal);
  const relevantHtml = await extractRelevantHtml(html, relevantElements);
  const gptResponses: string[] = [];

  for (const chunk of relevantHtml) {
    const prompt = `Given the goal "${goal}", analyze the following HTML chunk and provide a list of instructions to interact with the elements in a JSON format. Each instruction should have an "action" (e.g., "click", "type", "typeEnter", "extractText") and a "selector" (e.g., "#elementId", ".elementClass"). Ensure the instructions do not conflict with each other:\n\n${chunk}`;

    try {
      const gptResponse = await simpleSingleQueryGPTCall(prompt);
      if (gptResponse.trim() !== '') {
        gptResponses.push(gptResponse);
      }
    } catch (error) {
      console.error('Error calling GPT-3:', error);
    }
  }

  // Aggregate and process the GPT-3 responses
  const instructions = gptResponses.map((response) => JSON.parse(response));
  console.log('Instructions:', instructions);

  // Execute the instructions using Puppeteer
  await executeInstructions(instructions);
}

async function executeInstructions(instructions: any[]): Promise<void> {
  const puppeteer = require('puppeteer');
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  for (const instruction of instructions) {
    const action = instruction.action;
    const selector = instruction.selector;

    switch (action) {
      case 'click':
        await page.click(selector);
        break;
      case 'type':
        const text = instruction.text;
        await page.type(selector, text);
        break;
      case 'typeEnter':
        await page.type(selector, String.fromCharCode(13));
        break;
      case 'extractText':
        const extractedText = await page.$eval(selector, (el) => el.textContent);
        console.log('Extracted text:', extractedText);
        break;
      default:
        console.error('Unknown action:', action);
    }
  }

  // Close the browser after executing the instructions
  await browser.close();
}

const html = "<html><head><title>Example</title></head><body><form><input type='text' id='address-search' placeholder='Enter address'><button type='submit' id='search-btn'>Search</button></form><div id='median-income'>The median income in this neighborhood is $75,000.</div></body></html>";
const goal = 'This page has info on the median income in this neighborhood, give me that information';
processHtmlInChunks(html, goal);

In this updated version, the GPT-3 prompt includes the "extractText" action. The executeInstructions function has also been updated to handle the "extractText" action, which extracts the text content of the specified HTML element using Puppeteer's page.$eval method. The extracted text is then logged to the console.

does the processHtmlInChunks return the text I want for the goal?

now make everything real code with all the imports needed and also for pupeteer use browserless.io

// scrapes posts from hackernews
import puppeteer from "puppeteer";

// Try changing this! 👇
const keyWord = "";

const main = async () => {
  const browser = await puppeteer.connect({
    browserWSEndpoint: `wss://chrome.browserless.io`,
  });
  const page = await browser.newPage();
  await page.goto("https://news.ycombinator.com/");

  const data = await page.evaluate((keyWord) => {
    let $$news = [...document.querySelectorAll(".athing")].filter((elem) =>
      elem.querySelector(".votearrow")
    );

    if (keyWord) {
      $$news = $$news.filter(($news) =>
        $news
          .querySelector(".titleline")
          .innerText.toLowerCase()
          .includes(keyWord)
      );
    }

    return $$news.map((elem) => {
      const $info = elem.nextElementSibling;

      return {
        title: elem.querySelector(".titleline").innerText,
        url: elem.querySelector(".titleline a").href,
        score: parseInt($info.querySelector(".score").innerText),
        date: $info.querySelector("span.age").title,
        by: $info.querySelector(".hnuser").innerText,
      };
    });
  }, keyWord);

  return { data };
};

above is an example use of browserless

Here's the complete code with all the necessary imports and using browserless.io for Puppeteer:

import { simpleSingleQueryGPTCall } from './your_example_module';
import puppeteer from 'puppeteer';

async function identifyRelevantElements(html: string, goal: string): Promise<string[]> {
  const prompt = `Given the goal "${goal}", identify the relevant HTML elements (e.g., input, button, form, div, span) that should be considered for generating instructions to achieve the goal. The HTML content is as follows:\n\n${html}`;
  const gptResponse = await simpleSingleQueryGPTCall(prompt);

  // Extract the relevant elements from the GPT-3 response
  const relevantElements = gptResponse.split(',').map((element) => element.trim());
  return relevantElements;
}

async function extractRelevantHtml(html: string, relevantElements: string[]): Promise<string[]> {
  const parser = new DOMParser();
  const doc = parser.parseFromString(html, 'text/html');
  const relevantHtml: string[] = [];

  relevantElements.forEach((element) => {
    const elements = doc.querySelectorAll(element);
    elements.forEach((el) => {
      relevantHtml.push(el.outerHTML);
    });
  });

  return relevantHtml;
}

async function processHtmlInChunks(html: string, goal: string): Promise<string> {
  const relevantElements = await identifyRelevantElements(html, goal);
  const relevantHtml = await extractRelevantHtml(html, relevantElements);
  const gptResponses: string[] = [];

  for (const chunk of relevantHtml) {
    const prompt = `Given the goal "${goal}", analyze the following HTML chunk and provide a list of instructions to interact with the elements in a JSON format. Each instruction should have an "action" (e.g., "click", "type", "typeEnter", "extractText") and a "selector" (e.g., "#elementId", ".elementClass"). Ensure the instructions do not conflict with each other:\n\n${chunk}`;

    try {
      const gptResponse = await simpleSingleQueryGPTCall(prompt);
      if (gptResponse.trim() !== '') {
        gptResponses.push(gptResponse);
      }
    } catch (error) {
      console.error('Error calling GPT-3:', error);
    }
  }

  // Aggregate and process the GPT-3 responses
  const instructions = gptResponses.map((response) => JSON.parse(response));
  console.log('Instructions:', instructions);

  // Execute the instructions using Puppeteer and browserless.io
  const result = await executeInstructions(instructions);
  return result;
}

async function executeInstructions(instructions: any[]): Promise<string> {
  const browser = await puppeteer.connect({
    browserWSEndpoint: 'wss://chrome.browserless.io',
  });
  const page = await browser.newPage();
  let extractedText = '';

  for (const instruction of instructions) {
    const action = instruction.action;
    const selector = instruction.selector;

    switch (action) {
      case 'click':
        await page.click(selector);
        break;
      case 'type':
        const text = instruction.text;
        await page.type(selector, text);
        break;
      case 'typeEnter':
        await page.type(selector, String.fromCharCode(13));
        break;
      case 'extractText':
        extractedText = await page.$eval(selector, (el) => el.textContent);
        break;
      default:
        console.error('Unknown action:', action);
    }
  }

  // Close the browser after executing the instructions
  await browser.close();
  return extractedText;
}

const html = "<html><head><title>Example</title></head><body><form><input type='text' id='address-search' placeholder='Enter address'><button type='submit' id='search-btn'>Search</button></form><div id='median-income'>The median income in this neighborhood is $75,000.</div></body></html>";
const goal = 'This page has info on the median income in this neighborhood, give me that information';
processHtmlInChunks(html, goal).then((result) => {
  console.log('Result:', result);
});

In this code, the executeInstructions function has been updated to use browserless.io for Puppeteer. The processHtmlInChunks function now returns the extracted text as the output. The result is logged to the console after calling processHtmlInChunks.

you should allow gpt3 to not give an instruction if a chunk isn't relevant

To allow GPT-3 to skip providing an instruction if a chunk isn't relevant, you can modify the prompt to include this option. Here's an updated version of the processHtmlInChunks function that incorporates this change:

async function processHtmlInChunks(html: string, goal: string): Promise<string> {
  const relevantElements = await identifyRelevantElements(html, goal);
  const relevantHtml = await extractRelevantHtml(html, relevantElements);
  const gptResponses: string[] = [];

  for (const chunk of relevantHtml) {
    const prompt = `Given the goal "${goal}", analyze the following HTML chunk and provide a list of instructions to interact with the elements in a JSON format. Each instruction should have an "action" (e.g., "click", "type", "typeEnter", "extractText") and a "selector" (e.g., "#elementId", ".elementClass"). If the chunk is not relevant, return an empty response. Ensure the instructions do not conflict with each other:\n\n${chunk}`;

    try {
      const gptResponse = await simpleSingleQueryGPTCall(prompt);
      if (gptResponse.trim() !== '') {
        gptResponses.push(gptResponse);
      }
    } catch (error) {
      console.error('Error calling GPT-3:', error);
    }
  }

  // Aggregate and process the GPT-3 responses
  const instructions = gptResponses.map((response) => JSON.parse(response));
  console.log('Instructions:', instructions);

  // Execute the instructions using Puppeteer and browserless.io
  const result = await executeInstructions(instructions);
  return result;
}

In this updated version, the GPT-3 prompt asks to return an empty response if the chunk is not relevant. The processHtmlInChunks function will only add non-empty responses to the gptResponses array. This way, GPT-3 can skip providing an instruction for irrelevant chunks.

if a past chunk has something relevant you should include that in the prompt for the next chunk/s to tell gpt about past relevant chunks

then gpt should figure out how to reconcile the two chunks and instructions, either it should throw out one chunk or combine the html and instructions for both if both are relevant e.g. if you need to click on multiple places to accomplish a task and the places are spread across different chunks

you should also tell gpt3 shorten this reconciled chunk to only the minimum tags required and fix any broken tags

also the new reconciled chunk and instructions should be fed into every subsequent chunk so that at the end a final instructions is produced with a final relevant chunk (that could be a combination of multiple chunks)

To achieve this, you can modify the processHtmlInChunks function to keep track of the past relevant chunks and their corresponding instructions. You can then include this information in the prompt for the next chunk, allowing GPT-3 to reconcile the chunks and instructions.

Here's an updated version of the processHtmlInChunks function that incorporates these changes:

async function processHtmlInChunks(html: string, goal: string): Promise<string> {
  const relevantElements = await identifyRelevantElements(html, goal);
  const relevantHtml = await extractRelevantHtml(html, relevantElements);
  let pastRelevantChunks = '';
  let pastInstructions = '';

  for (const chunk of relevantHtml) {
    const prompt = `Given the goal "${goal}", analyze the following HTML chunk:\n\n${chunk}\n\nConsider the past relevant chunks:\n\n${pastRelevantChunks}\n\nAnd their corresponding instructions:\n\n${pastInstructions}\n\nReconcile the chunks and instructions, either by throwing out one chunk or combining the HTML and instructions for both if both are relevant. Shorten the reconciled chunk to only the minimum tags required and fix any broken tags. Provide a list of instructions to interact with the elements in a JSON format. Each instruction should have an "action" (e.g., "click", "type", "typeEnter", "extractText") and a "selector" (e.g., "#elementId", ".elementClass"). If the chunk is not relevant, return an empty response. Ensure the instructions do not conflict with each other.`;

    try {
      const gptResponse = await simpleSingleQueryGPTCall(prompt);
      if (gptResponse.trim() !== '') {
        const response = JSON.parse(gptResponse);
        pastRelevantChunks = response.relevantChunk;
        pastInstructions = JSON.stringify(response.instructions);
      }
    } catch (error) {
      console.error('Error calling GPT-3:', error);
    }
  }

  // Execute the final instructions using Puppeteer and browserless.io
  const finalInstructions = JSON.parse(pastInstructions);
  console.log('Final Instructions:', finalInstructions);
  const result = await executeInstructions(finalInstructions);
  return result;
}

In this updated version, the GPT-3 prompt includes the past relevant chunks and their corresponding instructions.

listeven989 / real-estate-deal-finder

parser for niche site #1