Updates to dev UI

[gorkem]

Describe the problem you're trying to solve Dev UI should be more helpful with helping the application developers integrate with the model.

Describe the solution you'd like

• Generate/show example code as the parameters and prompts are entered.
• A way to see JSON communicated between the server and responses
• hide or drop the list of preferences to highlight more frequently used ones
• Ability to see the README.md from the ModelKit if it has one

[annigro]

https://build.nvidia.com/explore/discover

[annigro]

UX Design

[gorkem]

Here is some more clarification on the requests.

POST /completion is an API specific to llama.cpp server. There is example code available for its usage which we will adjust for code generation feature.

POST /v1/chat/completions The endpoint for chat completion API is compatible with the OpenAI endpoint. We can use the existing OpenAPI libraries for code generation.

Options

The initial thought was to reduce the options to match OpenAI. However, with further thinking and considering the options are supported by both endpoints. We should continue to use all options but categorize them better.

Categories

Text Generation Controls

• Temperature: Controls randomness.
• Top_k: Limits token selection to the most probable.
• Top_p: Limits token selection by cumulative probability.
• Min_p: Sets a minimum probability threshold.
• N_predict: Max tokens to predict. (This could be a category of its own)

Sampling and Diversity

• Tfs_z: Tail-free sampling to control diversity.
• Typical_p: Locally typical sampling for natural text.
• Presence_penalty, frequency_penalty: Adjusts likelihood based on token presence and frequency.
• Mirostat, mirostat_tau, mirostat_eta: Controls complexity and randomness.
• Repeat_penalty, repeat_last_n: Controls repetition.

Advanced Settings and Customization

• Grammar, json_schema: Enables grammar-based sampling with JSON schema.
• api_key

Probability and Statistical Controls

• N_probs: Provides probabilities for top N tokens.
• Min_keep: Ensures a minimum number of tokens are returned.

single or multiple page

I have not really found a good reason to keep the multiple page. It is unintuitive to go back and forth for changing the parameters. I suggest we do a single page implementation.

[annigro]

--> We talked about how the generated code for chat mode can get really long and therefore messy. @gorkem I assume the first two lines are your solution. How would it look in the UI?

[javisperez]

@annigro no, Gorkem's first two lines are related to internal code usage, and should be transparent for the UI other than one line. We talked about doing something like this for it:

message: [{ actor: 'user', content: 'foo bar fooz' }]

and when is too long:

message: [{ actor: 'user', content: 'foo bar ...' }] <-- ellipsis but for the message's content only, and only if is too long.

but clicking on "copy code" would always copy the whole thing, regardless of the ellipsis.

[javisperez]

@gorkem i gave a try to the open ai api but looks like both the payload and the response is different than the one we already have in llama.cpp (which makes sense). I vote to keep using the completion llama.cpp endpoint instead or redoing everything including the api layer just to support openai v1 endpoints. Thoughts?