Handling Ground Truths

[0x4007]

4. Handling Ground Truths: They are indicating that the system uses “ground truths” — meaning predefined correct examples or comments that the system relies on for determining context. Even if the query (or comment) doesn’t provide enough context, the system tries not to make assumptions. For example, if the query asks about “types” in a code snippet without specifying a language, the system shouldn’t assume it’s referring to Python.

Hard coding these things is the wrong approach then. This needs to be dynamic in a new task.

_Originally posted by @0x4007 in https://github.com/ubiquity-os-marketplace/command-ask/pull/2#discussion_r1804155667_

A small collection of keywords doesn't seem like useful "ground truths" I would expect them to be larger bodies of text as each array element.

These should be derived dynamically. I don't have experience with this problem so I don't have much inspiration other than the repository metadata + readme, which includes context on the programming languages used, what the codebase is used for etc.

[sshivaditya2019]

@UbiquityOS are you alive now?

[ubiquity-os-beta[bot]]

Yes, I’m active and running smoothly. You can interact with me using commands like /myid to get your unique identifier or to set up admin permissions.

[sshivaditya2019]

@UbiquityOS can you tell me about UUSD ?

[sshivaditya2019]

@UbiquityOS are you alive now?

[ubiquity-os-beta[bot]]

Hello! Yes, I’m up and running. If you need assistance or want to check your ID, feel free to use the /myid command.

[gentlementlegen]

Seing these comments make me think that in the conversation rewards, we ignore commands but since these do not look like our usual commands they would be taken into the final reward, is that something we want?

[sshivaditya2019]

@UbiquityOS can you tell me about conversation rewards?

[0x4007]

Seing these comments make me think that in the conversation rewards, we ignore commands but since these do not look like our usual commands they would be taken into the final reward, is that something we want?

I think the simplest solution is to add an exception for if we are tagging the bot username at the beginning of the comment[^1^]. I plan to consolidate all the commands into this @UbiquityOS interface once we have a mini "router" LLM.

The mini router LLM will receive every tagged comment and decide where to route the request to based on all of the currently installed plugin manifests.

[^1^]: I wonder if it makes sense to exclude every comment with a direct tag in the beginning. I suspect that this implies you are not speaking in general, but instead requesting something specific from another contributor. Maybe it makes sense to exclude from rewards? All I know is that I never start a comment in this format except for when trying to talk to the bot. Instead I ask first then rfc, for whatever its worth.

[gentlementlegen]

@0x4007 The bot's username can be different for each organization technically, and actually tagging @UbiquityOS is tagging the organization url in the comment. How do we know a user tagged the bot?

[0x4007]

Hard code that as the bot name

No every org is to use our official app, and theres only one for prod, one for dev

I suppose we can consider catching all related tags like

@ubiquity-os @ubiquity-os[bot] etc

[gentlementlegen]

Then it should be better to export something from the SDK so every plugin is using the same name.

[0x4007]

No the kernel should route these requests. The plugins should retain their direct / invocation syntax because its concise to say

/start

Instead of

@ubiquityos can you assign me