[Feature Request]: Implementation for WebChatGPT

[LuckyHead11]

Is there an existing issue for this?

• [x] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

I have just come across the GitHub page: https://github.com/qunash/chatgpt-advanced ChatGPT-Advanced or WebChatGPT. The title is self-explanatory and gives ChatGPT access to the web similar to what BingGPT does through a Chrome Extension. I would like to know if you can implement something like it. This is what it does:

1. Searches google and looks for relevant info.
2. Changes the user's prompt to something like this: With the prompt being What is the ChatGPT Github page by acheong all about?

You can ask directly to @qunash if you can implement or share code. Or you can copy the idea and make a similar feature to the API.

Proposed workflow

When you initialize the ChatBot in python you would simply add a "web" parameter. Which could be equal to True or False.

Additional information

No response

[acheong08]

It'll take some time. I'll try to implement this next week when my mock exams are over

[acheong08]

@f1nniboy has something similar to this as well. I'll ask him to share source code.

[LuckyHead11]

Alright thank you for considering!

[wlc002]

good idea!

[LuckyHead11]

Thanks! I didn't really come up with the idea for WebChatGPT at first. I just thought it would be cool to see it here.

[boxiyang]

pynecone has something similar to the request, with local users and questions/answers management. You can take it as a reference. https://github.com/pynecone-io/pynecone-examples/tree/main/gpt

[noseon]

I just reverse engineered the plugin and switched to python below. you can put it in code with a simple function. https://github.com/noseon/WebChatGPT-/blob/main/WebChatGPT%20.py

[LuckyHead11]

@noseon that is very impressing!

[NymdaFromSalad]

Haha, i also reverse-engineered it for my project. Should've looked into issues earlier, could've been first :)

[LuckyHead11]

@NymdaFromSalad is it on a GitHub repo?

[acheong08]

I just reverse engineered the plugin and switched to python below. you can put it in code with a simple function. noseon/WebChatGPT-@main/WebChatGPT%20.py

I would rather not use a third party API. I wrote my own DuckDuckGo scraper API today in Go: https://github.com/acheong08/DuckDuckGo-API

[acheong08]

As a side note, this will probably be a separate repository rather than part of this one.

I feel that an API would be most suitable for this rather than a Python library

[noseon]

pass your code to python and use it as a secondary api to your ChatGPT.

[LuckyHead11]

Sounds great!

[NymdaFromSalad]

@LuckyHead11 No. It's here :) :

import requests, time

def apiSearch(query: str, numResults: int, timePeriod: str='', region: str=''):

    searchParams = ''
    searchParams += f'q={query}'
    searchParams += f'&max_results={numResults}'
    if timePeriod:
        searchParams += f'&time={timePeriod}'
    if region:
        searchParams += f'&region={region}'

    url = f'https://ddg-webapp-aagd.vercel.app/search?{searchParams}'

    response = requests.get(url)

    results = response.json()

    return results

def prepare_results(results):
    i=1
    res = ""
    for result in results:
        res+=f'[{i}] "{result["body"]}"\nURL: {result["href"]}\n\n'
        i+=1
    return res

def headers(results):
    res = []
    for result in results:
        #res.append({'text':f'[{i}] {result["title"]}', 'link':result["href"]})
        res.append(result["href"])
    return res

def compile_prompt(prompt, results, reply_in = 'language of the query'):
    return f'''Web search results:

{prepare_results(results)}
Current date: {time.strftime("%d.%m.%Y")}

Instructions: Using the provided web search results, write a comprehensive reply to the given query. Make sure to cite results using [number] notation after the reference. If the provided search results refer to multiple subjects with the same name, write separate answers for each subject.
Query: {prompt}
Reply in {reply_in}'''

def Webify(text:str, query: str, numResults: int, timePeriod: str='', region: str='', reply_in: str='undefined'):
    searchResults = apiSearch(query, numResults, timePeriod, region)
    return compile_prompt(text, searchResults, reply_in), headers(searchResults)

if __name__ == '__main__':
    print(Webify('Testing', 3))

text and query are separate, 'cause i also wanted to do other filters, which can add instructions

Also if you wandering, why the function is called headers, that's because in my native language, headlines, titles and headers means the same

[LuckyHead11]

Awesome! @NymdaFromSalad @acheong08 with the code provided by Nymda implementing the Web/ChatGPT mix should be easy. Unless you would like to create your own way of using the web with ChatGPT ;) As you said though, you wouldn't use a 3rd party API.

[noseon]

@LuckyHead11Não. Está aqui :) :

import requests, time

def apiSearch(query: str, numResults: int, timePeriod: str='', region: str=''):

    searchParams = ''
    searchParams += f'q={query}'
    searchParams += f'&max_results={numResults}'
    if timePeriod:
        searchParams += f'&time={timePeriod}'
    if region:
        searchParams += f'&region={region}'

    url = f'https://ddg-webapp-aagd.vercel.app/search?{searchParams}'

    response = requests.get(url)

    results = response.json()

    return results

def prepare_results(results):
    i=1
    res = ""
    for result in results:
        res+=f'[{i}] "{result["body"]}"\nURL: {result["href"]}\n\n'
        i+=1
    return res

def headers(results):
    res = []
    for result in results:
        #res.append({'text':f'[{i}] {result["title"]}', 'link':result["href"]})
        res.append(result["href"])
    return res

def compile_prompt(prompt, results, reply_in = 'language of the query'):
    return f'''Web search results:

{prepare_results(results)}
Current date: {time.strftime("%d.%m.%Y")}

Instructions: Using the provided web search results, write a comprehensive reply to the given query. Make sure to cite results using [number] notation after the reference. If the provided search results refer to multiple subjects with the same name, write separate answers for each subject.
Query: {prompt}
Reply in {reply_in}'''

def Webify(text:str, query: str, numResults: int, timePeriod: str='', region: str='', reply_in: str='undefined'):
    searchResults = apiSearch(query, numResults, timePeriod, region)
    return compile_prompt(text, searchResults, reply_in), headers(searchResults)

if __name__ == '__main__':
    print(Webify('Testing', 3))

texto e consulta são separados, porque eu também queria fazer outros filtros, que podem adicionar instruções

Além disso, se você está vagando, por que a função é chamada de cabeçalhos, é porque no meu idioma nativo, manchetes, títulos e cabeçalhos significam o mesmo

and that's how it works

[acheong08]

Also need a mechanism for it to detect whether to search or not and convert natural language to a search query

[NymdaFromSalad]

Also need a mechanism for it to detect whether to search or not and convert natural language to a search query

Well, that's natural language processing and it is hard. Simplest solution would be to require user press the button and either using the whole query as the search parameter, or asking user for a search term. Honestly, forgot to check if WebGPT has any other mechanism. If it does - well, i need to reverse engineer it either way.

Also can use keywords, like putting /search at the start, makes it use WebGPT

Also you can ask ChatGPT to do NLP, as it is NLM :)

[NymdaFromSalad]

and that's how it works

Also must mention: 1) Search app gets content and cuts it to a certain size. (Maybe article preview feature can be used to extract data?) 2) In the OG ChatGPT (website chat), WebGPT makes it use markdown and put references as (url)[number]. 3) For some reason WebGPT, when you select default profile, asks to 'Reply in Undefined' 4) Headers function is used to put links into the final message, as API returns plaintext (and refs turn into just [number]) you can modify it, depending on how you want to put references in

[LuckyHead11]

Also need a mechanism for it to detect whether to search or not and convert natural language to a search query

we could use a different package, like for V1: revChatGPT.V1 But for the web access one would be revChatGPT.V1_WEB or when you do chatbot.ask it would have a web parameter as I said in the beginning.

[NymdaFromSalad]

Also need a mechanism for it to detect whether to search or not and convert natural language to a search query

we could use a different package, like for V1: revChatGPT.V1 But for the web access one would be revChatGPT.V1_WEB or when you do chatbot.ask it would have a web parameter as I said in the beginning.

Oh, in that sense? Then:

0) Needs better name, .webaccess .web or .webgpt are my picks 1) You can create a web-accessing bot class, and either pass it chatbot object, or create internal chatbot instances, then it'll have additional functions to process web stuff and retain the OG ones

I dunno, i have some time on the weekend (and need to take a break from my personal project's SQL hijinx), if you guys want, i can make it (and do something i never done before - write decent code)

(Also, will try and use duckduckgo, as putting feature, that uses other people's (who do opensource, might i add) private servers without their consent into a library is not very nice. We only do it to evil corporations)

[LuckyHead11]

Also need a mechanism for it to detect whether to search or not and convert natural language to a search query

we could use a different package, like for V1: revChatGPT.V1 But for the web access one would be revChatGPT.V1_WEB or when you do chatbot.ask it would have a web parameter as I said in the beginning.

Oh, in that sense? Then:

0. Needs better name, .webaccess .web or .webgpt are my picks
1. You can create a web-accessing bot class, and either pass it chatbot object, or create internal chatbot instances, then it'll have additional functions to process web stuff and retain the OG ones

I dunno, i have some time on the weekend (and need to take a break from my personal project's SQL hijinx), if you guys want, i can make it (and do something i never done before - write decent code)

(Also, will try and use duckduckgo, as putting feature, that uses other people's (who do opensource, might i add) private servers without their consent into a library is not very nice. We only do it to evil corporations)

Good Ideas! It's up to @acheong08 if you should make it. EDIT: Acheong did mention making another repository in that case WebGPT would be a good name for that.

[NymdaFromSalad]

Well, if you guys wanna use it, either way, it'll be time well spent

[NymdaFromSalad]

EDIT: Acheong did mention making another repository in that case WebGPT would be a good name for that.

WebGPT is my slang for WebChatGPT, and it is also a name of another OpenAI product https://openai.com/research/webgpt Also, he was talking about an API to implement this feature, which is not my idea (as i see no benefit to putting it on a server. It is not a passive scraper, it only scrapes one query, on demand. The delay would be mostly due to page loading time anyway, and very minor too. Why not just do the scraping locally?)

[acheong08]

(Also, will try and use duckduckgo, as putting feature, that uses other people's (who do opensource, might i add) private servers without their consent into a library is not very nice. We only do it to evil corporations)

There is no need for that. I already wrote my own API for it and currently hosting on Heroku for testing https://github.com/acheong08/DuckDuckGo-API

[acheong08]

I dunno, i have some time on the weekend

Let me know if you do. There's no point in duplicating efforts.

[NymdaFromSalad]

(Also, will try and use duckduckgo, as putting feature, that uses other people's (who do opensource, might i add) private servers without their consent into a library is not very nice. We only do it to evil corporations)

There is no need for that. I already wrote my own API for it and currently hosting on Heroku for testing https://github.com/acheong08/DuckDuckGo-API

Well, using your api is one of two options i consider. As i said, there is no need to put the code on a server, if it can perfectly run locally imo. But also, there is no point to inventing bicycle again, so it is a question of 'is it worth it to translate a scraper, if it will use less servers'

Also i was referring to WebChatGPT's server in that fragment

But hey, if you're against me scraping locally in this situation, then ok, will use your api

[acheong08]

I prefer scraping locally

[acheong08]

I just wrote the API because I couldn't figure out how to scrape reliably locally in Python.

[acheong08]

I just misunderstood

[acheong08]

No way I want more traffic lol. I don't want to pay for that

[acheong08]

So are you gonna work on this?

[LuckyHead11]

I just wrote the API because I couldn't figure out how to scrape reliably locally in Python.

How is python unreliable?

[acheong08]

I just wrote the API because I couldn't figure out how to scrape reliably locally in Python.

How is python unreliable?

Before OpenAI removed captcha from auth0, there were a ton of issues with the bs4 parser, especially lxml. It just doesn't work on some machines and I have no idea why

[acheong08]

I ended up moving to just using regex but that's quite finicky

[LuckyHead11]

Ah, that makes sense. Will you be making a new repo or just edit this one?

[acheong08]

If @NymdaFromSalad is gonna make a good one, I would rather just like their repo in the readme. Otherwise, It'll probably be a new repo

[NymdaFromSalad]

So are you gonna work on this?

Yes

If @NymdaFromSalad is gonna make a good one, I would rather just like their repo in the readme. Otherwise, It'll probably be a new repo

I was going to make an addition to this repo. Like, i wanted to make a submodule, not a standalone project - i have enough of those

[NymdaFromSalad]

I was just made to play mafia (social game) for 3 hours straight, my brain is on cooldown, i am confused

[NymdaFromSalad]

Ok, i figured it out. Implementation tomorrow

[acheong08]

I added a quick implementation today. I'll publish release later

[NymdaFromSalad]

I added a quick implementation today. I'll publish release later

So, you already did it? Using your server or not?

[acheong08]

So, you already did it? Using your server or not?

A very basic version using my server. It sucks right now lol. It's just for testing

[NymdaFromSalad]

So, you already did it? Using your server or not?

A very basic version using my server. It sucks right now lol. It's just for testing

Oh ok, then my idea still stands

[acheong08]

Oh ok, then my idea still stands

yes.

[LuckyHead11]

Is the new Web accessed chatgpt only for V3? @acheong08

[NymdaFromSalad]

Ok, sorry for my disappearance. I am trying to get off caffeine and got a massive wave of tiredness, and slept all day. Working on V1 implementation RN, almost finished (editing wiki now). After this, will look into V3

[LuckyHead11]

Alright! Great, is there any way I could help?

[NymdaFromSalad]

Alright! Great, is there any way I could help?

Certainly! You can get the extension (or just imitate it, which would be even simpler) and try out different prompt templates for it

Note: it should mark references like this[num], or likenum. Since both turn to former format in the end, it is preferable (lil faster)

Also, when testing, it stopped putting direct references at all (no idea, why), so make sure it does reference it. Then, post/send me your results

Also also, try different language prompts (and search results)

This will massively help, since such models are all about prompt crafting