Add tokenizer app

[alonsosilvaallende]

This PR adds the GPT-4 token encoder and decoder application to the examples folder.

[maartenbreddels]

Thank for this, I think it's a really cool example.

I need to digest a bit on what is happening, and i'd like to do some refactoring.

Also, I'm getting this error:

{
    "error": {
        "message": "Incorrect API key provided: ---. You can find your API key at https://platform.openai.com/account/api-keys.",
        "type": "invalid_request_error",
        "param": null,
        "code": "invalid_api_key"
    }
}

Do you know why this is happening, because I don't think we are calling an API right?

[alonsosilvaallende]

Yes, feel free to refactor it. We are not calling the API so that error is weird indeed. Here is a colab which should work without errors: https://gist.github.com/alonsosilvaallende/218fc71775ea1056e9c0233737dad558

[maartenbreddels]

I've included this screenshot:

For a live screencapture, see: https://twitter.com/alonsosilva/status/1714352902242779145

[maartenbreddels]

Ok, that error was my fault, I had some gpt experiment app sitting in the same directory that triggered it.

[maartenbreddels]

Really nice, if we use #336, we can do the following:

-                Token(token)
+                token_element = Token(token)
+
+                with solara.lab.ClickMenu(activator=token_element):
+                    with solara.Column(gap="0px"):
+
+                        def copy_token(token=token):
+                            tokens_ids_to_lookup.set(tokens_ids_to_lookup.value + " " + str(token))
+
+                        solara.Button("Copy token ID", on_click=copy_token, text=True)
+
+                        def search(token=token):
+                            token_string = tokenizer.decode([token])
+                            tokens_filter.set(token_string)
+
+                        solara.Button("Search for token", on_click=search, text=True)

So we can add actions to each token:

[alonsosilvaallende]

That's awesome. I love it.

[maartenbreddels]

I didn't implement that in this PR, this was meant more as an example for you to use in other solara based apps. Or do you think we should push it to this branch?

[alonsosilvaallende]

Thank you. No, I think we don't need to push it to this branch. I will definitely use it in other solara based apps.

[maartenbreddels]

I've added a credits section (your name + avatar) to it. Let me know if you like it as it is now.

PS: the email address in the git logs were wrong, when rebasing, you don't get the credits. I've fixed it manually now, but I recommend you fix your local git configuration with the right email address.

[alonsosilvaallende]

Thank you very much. Yes, it's perfect. Thank you for the heads-up about the e-mail address. I have fixed it now.

[maartenbreddels]

Awesome, thank you !