Generic import format

[drusepth]

I'm looking into what it'd take to write an exporter for Notebook.ai that lets users either import individual characters or all of their characters into OpenCharacters. Being able to talk with the characters you've been building seems like an awesome feature for authors and other creatives who want to make their characters feel a little more real before, during, or after writing stories about them.

A lot of this is just thinking out loud and translating my interpretation of your code to words, but there are a few questions below too. :)

I'm using Zoltanai's character editor's JSON format as a template for a valid import format, which includes the following fields:

• char_name
• char_persona
• world_scenario
• char_greeting
• example_dialogue
• name
• description
• personality
• scenario
• first_mes
• mes_example

Looking at the exported values from the tool, it looks like a few of these fields are redundant to cover importing to different tools expecting various character formats, so I just want to make sure I'm reducing the set to what's actually used here and packing as much information into those keys.

Looking at tryImportingExternalCharacterFileFormat, it looks like the generic mapping is as follows:

Export field	maps to OpenCharacters field
name	name
avatar	avatarUrl
charPersona / personality	roleInstruction
description	description
scenario / world_scenario	scenario
None [1]	reminderMessage
char_greeting / first_mes	initialMessage
example_dialogue / mes_example	example_dialogue

[1] I don't see anything in the import code to set reminderMessage, but I do see it gets used elsewhere (and can be set from the UI). Is this something available for imports and I just missed where it's set?

It looks like just name, avatar, personality, description, scenario, example_dialogue, and char_greeting are the used keys for a full import. Am I missing anything?

I'll try to get a generic export prototyped in Notebook.ai soon so I can see how well it works OOTB with the current generic import functionality, but if there are other details that would be helpful to each chatbot to have and improve quality on your end, I'd definitely be open to exporting those too! :)

A couple extra questions (or maybe feature requests) on the import process also:

1. Does OpenCharacters support importing multiple characters at once, or is it always just one at a time? It'd be great to let users easily import all of their characters at once and just pick and choose which one to chat with from the standard "new chat" UI.
2. Is there an endpoint I can link to and code the export data to have it automatically import / populate character(s)? For example, it'd be great to have a "Talk to this character" link on my end that routes a user to OpenCharacters with that character's information and lets them immediately start chatting, as opposed to offering an export download to be re-uploaded as an import and then start chatting.

[josephrocca]

Is there an endpoint I can link to and code the export data to have it automatically import

This might be the easiest way for you to integrate - at least as an MVP. The character share links have a hash with some JSON embedded. It looks like this:

https://josephrocca.github.io/OpenCharacters/#%7B%22addCharacter%22%3A......

Here's some example JSON:

https://gist.github.com/josephrocca/f4232e5852543bf1bbd0d0c8c3971ffe

You can of course construct a URL like that with:

let obj = {
  name: "...",
  roleInstruction: "...",
  ...
};
let url = `https://josephrocca.github.io/OpenCharacters/#${encodeURIComponent({addCharacter:obj})}`

So you could just have a "Chat on OpenCharacters" link on the character card in your UI. Hmm - I should probably make it so that if the character already has an exactly-matching character, then it doesn't show the character editor - just goes straight to the chat. Or maybe an option like {addCharacter:obj, skipCreationIfAlreadyExists:true} - so people can jump straight to the chat from your UI if they've already created that character in OpenCharacters.

I think someone on the Discord wanted something like {updateCharacter:...} - not exactly sure how that would work (how does it know which character to update? just based on the name?), but let me know your thoughts here. Users probably don't want to create a whole new character in OpenCharacters every time they slightly update a character in your world building UI.

It'd be easy to allow importing a json file that that has an array of the {addCharacter:...} objects - let me know if you want me to implement that, and if there are any particular requirements that you have, or interesting things that I might not expect (to help me form the right abstractions for all this stuff)

[drusepth]

Awesome, thanks for the guidance! I've got a working proof-of-concept export that seems to be working well enough, so now I get to play with prompts and make sure enough of the character's personality passes through.

I think someone on the Discord wanted something like {updateCharacter:...} - not exactly sure how that would work (how does it know which character to update? just based on the name?), but let me know your thoughts here. Users probably don't want to create a whole new character in OpenCharacters every time they slightly update a character in your world building UI.

I wanted to give my users a bit of transparency and customizability for what details get included in the export, since most users will have so many details on each character that it makes sense to pick and choose only the most relevant details to include. To that end, I gave them a little pre-populated form to customize before exporting, which seems relevant here -- since they'll be able to tweak character details (in the form or on the rest of the site) and/or the scenario before re-exporting.

The ideal UX here, I think, is to be able to pass along some kind of constant, unique identifier or UUID for each character (that's passed in or created upon character creation/import, rather than being a hashed function of the character's name or data) so repeated exports will always update the same character (even if the name or other data has changed). I would hope this would prevent any accidental clobbering of existing characters in OpenCharacters if a user were to try to export a character with the same name as one of their existing-but-different characters.

With unique UUIDs, I think you can skip the addCharacter/updateCharacter branching (as well as extra flags like skipCreationIfAlreadyExists) and always default to updating a character with that ID, or create it if it doesn't exist yet.

It'd be easy to allow importing a json file that that has an array of the {addCharacter:...} objects - let me know if you want me to implement that, and if there are any particular requirements that you have, or interesting things that I might not expect (to help me form the right abstractions for all this stuff)

I'm gonna hold off on implementing multi-character imports for now because I think the flow of going from a character page to a dialogue with them is a lot cleaner than whisking users off to an export and giving them directions on how to import, but I might come back to it in the future if I transition back towards actual exports instead of character share links -- it looks like share links are a great, frictionless solution unless there are some limitations I haven't found yet!

[josephrocca]

Thanks! The UUID is a good idea I think. Maybe I'll put some requirements on it to reduce probability of accidental collisions - e.g. must contain 6 hyphens, and they must have some non-hyphen characters between them. If your database ids are already UUIDs, then you can of course just do ${yourDatabaseId}-0-0-0-0-0 - adding this requirement is just to force devs to put some thought into making it unique, instead of e.g. just assigning an integer ID from their database.

Let me know if you have any thoughts there, otherwise this should be implemented within the next couple of days.

[drusepth]

That sounds great! Requiring hyphens is a clever way to avoid people accidentally (or intentionally) overwriting existing characters without resorting to something more clunky like namespacing or authorization strategies. My only suggestion would be to only require 4 hyphens. Per Wikipedia: Universally unique identifier:

In its canonical textual representation, the 16 octets of a UUID are represented as 32 hexadecimal (base-16) digits, displayed in five groups separated by hyphens, in the form 8-4-4-4-12 for a total of 36 characters (32 hexadecimal characters and 4 hyphens).

I think you'll find that most native UUID implementations result in this format, and requiring devs to pad an extra -0-0 on the end to add in a few more hyphens might be unintuitive.

Ruby:

irb(main):002:0> SecureRandom.uuid
=> "5e4e9755-0f0f-4262-af47-c79f219033f2"

Python:

>>> import uuid
>>> uuid_str = print(uuid.uuid4())
7dd06cd4-4282-48b7-b745-cd2250da1214

Javascript:

import { v4 as uuidv4 } from 'uuid';
uuidv4(); // ⇨ '9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d'

Java:

UUID.randomUUID(); // => f1e5d1f5-247f-44a3-b9c2-7e776179f5b5

etc.

[josephrocca]

Okay, this is done now - just add uuid property to the addCharacter object:

let obj = {
  uuid: "...",
  name: "...",
  roleInstruction: "...",
  ...
};
let url = `https://josephrocca.github.io/OpenCharacters/#${encodeURIComponent({addCharacter:obj})}`

It uses the proper UUID format that you suggested. There might be bugs - please ping me if so. Thanks for your help in designing this feature!