Working with pixelhop on the Scripting chatbot

[scripting]

We started working out how we're going to approach it in an email thread, but felt it would be better to move it to GitHub so as to create a record.

Right now -- as this thread starts -- we've tried OPML, but that had problems, so I'm writing a script to convert the OPML to Markdown.

[zefman]

Hey @scripting excited to see how markdown improves things!

[scripting]

Here's the markdown version of the files for 2022.

http://scripting.com/publicfolder/chatgpt/scriptingNewsSource/markdown/2022.zip

In the meantime I'll build the markdown versions for the other years.

[zefman]

Here we go here's a version using the markdown content: https://chatthing.ai/bots/b5c0ad69-55d4-4008-a97c-0b2884cca3f2/

[scripting]

The rest of the first batch of years is there now.

http://scripting.com/publicfolder/chatgpt/scriptingNewsSource/markdown/2023.zip http://scripting.com/publicfolder/chatgpt/scriptingNewsSource/markdown/2022.zip http://scripting.com/publicfolder/chatgpt/scriptingNewsSource/markdown/2021.zip http://scripting.com/publicfolder/chatgpt/scriptingNewsSource/markdown/2020.zip http://scripting.com/publicfolder/chatgpt/scriptingNewsSource/markdown/2019.zip http://scripting.com/publicfolder/chatgpt/scriptingNewsSource/markdown/2018.zip http://scripting.com/publicfolder/chatgpt/scriptingNewsSource/markdown/2017part2.zip

[scripting]

I'm trying the chatthing you sent me. I assume this is for 2022, the markdown file I sent you first.

I asked two questions -- did i know about "dave winer" and did it know about drummer.

drummer is a product i shipped in 2022, so there should be a lot of info about drummer, but it just came back with my bio.

[scripting]

BTW are you spending a lot of money to compile this stuff? I assume you'll tell me when you need me to participate in that.

[zefman]

Oh repeating the same answer after you asked about drummer seems to be a bug. If I start a new chat I get the following:

We will look into that.

Thanks at some point we can transfer this bot to your account but at the moment this is great testing for us!

[scripting]

Very nice. Where did it get the info about users? I don't think I posted any negative reviews to Scripting News.

Let me know when you have the other years loaded, I have more questions. ;-)

BTW -- I was sure Scripting News would make a good test case for this stuff, because the archive goes back so far and includes lots of stuff from other blogs, esp at the beginning. It was the hub for the early blogosphere.

I think researchers in general will find it's a strong use-case if not reference source. I was very conscious of this as the years went by, and tried to make it an archive not just for my work but for what was going on around it too.

[scripting]

I asked if when Drummer shipped, and it said it hadn't shipped yet. Because it only knows about July 2022. ;-)

Then I asked how Dave felt about Twitter...

[scripting]

BTW, one really important feature would be to have it make a static html transcript available, so we don't have to do screen shots for everything. I have something that does that for ChatGPT, but I really need it everywhere. ;-)

[scripting]

I have one big archive with all the stuff I have from 1994 to 2023. All converted to markdown, and organized by year, with a separate folder for DaveNet, which is how it all started in 1994. You should just import all of it. And that will be the product.

Here's the file, it's 12.3MB.

http://scripting.com/publicfolder/chatgpt/scriptingNewsSource/scriptingInMarkdown.zip

Can't wait to be able to play with this. ;-)

[zefman]

Amazing thanks Dave, will get this in in a minute!

We actually have the functionality to view previous conversations its just not surfaced in the UI yet. For example: https://chatthing.ai/bots/b5c0ad69-55d4-4008-a97c-0b2884cca3f2/chats/54392679-165d-4b09-b410-cfb3ba22759a/

[scripting]

I love it. You asked the right question about Web 3. This is better than a memoir! :-)

I think my mind is going to move into this space, real soon.

We should do a podcast. I'd like to understand what's running on your server, and how you did all this. I have only a user's perspective on this stuff, and barely that. Have been using ChatGPT and having my mind blown almost daily at how useful it is. Now this should be a whole new level.

[zefman]

Hey @scripting the whole lot is uploaded now!

This is by far the biggest bot we have tested with so we will be interested to see how it performs. It has so much data to play with now I wonder if it will have issues surfacing the most relevant content. From our test using the messages we sent earlier it still seems to be working well but can't wait to see what you find.

[scripting]

Already having fun.

A couple of things...

1. I want more vertical space, since I have to use screen shots for now to share.
2. Wouldn't it be cool if each user had an RSS feed of their queries, then we could build any kind of system we wanted around it.
3. I asked it about Larry Tesler, a former exec at Apple, who died. I asked "What does Dave say about Larry Tesler?" I'm not sure but I think it made up the last part of the answer. It doesn't sound like something I say. Will that happen here?

[scripting]

I posted items on Twitter, Mastodon and my blog.

1. https://twitter.com/davewiner/status/1640819380719058949
2. https://mastodon.social/@davew/110102913358313175
3. http://scripting.com/2023/03/28.html

[zefman]

I love it. You asked the right question about Web 3. This is better than a memoir! :-)

I think my mind is going to move into this space, real soon.

We should do a podcast. I'd like to understand what's running on your server, and how you did all this. I have only a user's perspective on this stuff, and barely that. Have been using ChatGPT and having my mind blown almost daily at how useful it is. Now this should be a whole new level.

Yes its very captivating isn't it! Hard not to be absorbed by it all what an exciting time!

We would love to explain to you how it works. Of course we are standing on the shoulders of giants, most of our work is just building a system around OpenAI apis!

[zefman]

Already having fun.

A couple of things...

1. I want more vertical space, since I have to use screen shots for now to share.
2. Wouldn't it be cool if each user had an RSS feed of their queries, then we could build any kind of system we wanted around it.
3. I asked it about Larry Tesler, a former exec at Apple, who died. I asked "What does Dave say about Larry Tesler?" I'm not sure but I think it made up the last part of the answer. It doesn't sound like something I say. Will that happen here?

Yes I agree with the vertical space, we will make it fill the available height.

Love the RSS idea, thats something we can look into as well.

So in terms of it making things up, that can happen! The main way we can prevent that is by adjusting the bot "temperature". This is basically a value between 0 and 1 that determines how creative the model is with its responses. Low temperatures make the responses close to deterministic, high temperatures add a lot more randomness and allow the model to fill in gaps.

[akaKenSmith]

If I were working with this tool on my own collected writing, I'd be interested in each message from the app ending with a sentence like this: "This reply was composed with the bot temperature set on [0.1, 0.6, 1.0, whatever it was]." Or just "Bot temp: 0.6"

[scripting]

A question follows...

I asked "When did Dave first write about the iPhone?"

What's interesting is that it provided the date of the first piece, Jan 10, 2007. How did it know the date? The Markdown files organize things by date, did the algorithm infer the date from the structure, or did the pixelhop folks write code to tell it what the date was?

Here's the HTML archive for the day.

http://scripting.com/2007/01/10.html

I did write about the iPhone on that day, but I don't know that it was the first reference.

[zefman]

A question follows...

I asked "When did Dave first write about the iPhone?"

What's interesting is that it provided the date of the first piece, Jan 10, 2007. How did it know the date? The Markdown files organize things by date, did the algorithm infer the date from the structure, or did the pixelhop folks write code to tell it what the date was?

Here's the HTML archive for the day.

http://scripting.com/2007/01/10.html

I did write about the iPhone on that day, but I don't know that it was the first reference.

So the data is included in plain text in the Markdown files so it should be able to use that to understand when you wrote about things.

The question about whether this is actually the first time you wrote about the iPhone is a lot harder to answer. The bot pulls in parts of your blog based on their semantic relevance to your question. I'm not sure if it takes into account when you say "first" that older dates would be more relevant. I would be interested to hear an OpenAI employees opinion on this!

[scripting]

@zefman --

This is how you can get a Markdown version of my blog's RSS feed.

http://feeder.scripting.com/returnmarkdown?feedurl=http%3A%2F%2Fscripting.com%2Frss.xml

It's brand new, nothing depends on it, so I can easily make changes. I designed it to follow the basic pattern of the OPML files.

[scripting]

@zefman -- you guys have been great. wanted to be sure people knew about your work. thanks!

[zefman]

Thanks for the shoutout @scripting its really appreciated! Just replying to your email now.

[drewkime]

Is there a way to have it output links to go with the references?