planning: Jan + Cortex Extensions UI Framework

[imtuyethan]

Goal

• Jan App Screens are Extension API-driven

Components

• Register New Settings Page
• Register New Icon in Left Ribbon
• Register New Tab in Right Menu
• Register New Input Modality in Input Box

Tasklist

• [x] https://github.com/janhq/jan/issues/3789

[dan-homebrew]

Goal

• Define what "UI" components can be configured by Extensions
  • e.g. Provider Extension
  • e.g. Tool Extension
  • e.g. System Monitor Extension
• Key Areas
  • Input Box
  • Ribbon
  • Right Panel
• What are some interesting cases
  • RAG Tool -> Attachments
  • Speech Model -> Microphone shows up in the input box
  • Provider Extension -> register Settings, register Params in right panel

Resources

• https://docs.obsidian.md/Plugins/User+interface/About+user+interface

[imtuyethan]

First design draft: https://www.figma.com/design/DYfpMhf8qiSReKvYooBgDV/Jan-App-(3rd-version)?node-id=8658-155262&t=1ac4R3yj6HBlxKRQ-4

[dan-homebrew]

Remote API Extensions:

• Registers Inference Settings (e.g. context length = 8192)
• Registers Settings page

[dan-homebrew]

Tools Extensions:

• Register Tools Extension settings on Right Panel (collapsible)
• Input Box: Tool Selector
• File Search
  • "Searching" status bar (in the UI)
  • Register "sources" in the response
• Web Search
  • "Searching" in progress (in the UI)
  • If we want a Perplexity-like structure, it would register "sources" in the response

[imtuyethan]

Here is the modular, extensible UI architecture for Jan, inspired by frameworks like Obsidian and VS Code:

Notes:

• This UI framework ensures that extensions can seamlessly modify or add UI components in the specified areas
• Jan Mobile should mirror the Desktop experience as closely as possible, preserving the configurable, extension-friendly UI layout #3905

---

By Layout

(1) Top Bar - Control panel

Always visible and acts as the central hub. It provides users with quick access to essential actions for different spaces and overall app settings. Users can easily navigate and execute commands using the command palette or control toolbar.

(2) Ribbon

• Always remains visible, functions as a surface for users’ spaces. Each space is represented by an icon in the ribbon.
• Top set of icons: includes threads & threads complementary spaces/actions like Hub, fine-tuning studio. Any spaces/actions added by extensions will be shown here.
• Bottom set of icons: houses essential app system actions or separate modules such as settings or local API servers.

(3) Status Tray

• Located at the bottom of the interface, spans full width and accommodates slots for extensions to register.
• Here, users can easily monitor vital updates such as model status (e.g., download, import) and relevant app status information.

(4) Content space

• Serves as the primary area displaying the main content of the page, its layout dynamically adjusting based on each page's requirements.
• Optionally, it may include left and right panels to supplement additional functions tailored to the specific needs of each page (e.g. Artifacts feature)
• It is designed to draw users' attention, positioned as a higher surface layer on top of the base interface.

1. Top bar

(1.1) Toolbar

• Includes quick access to essential actions to adjust app layout or settings Show only when contextually appropriate
• Developers can either choose to place icons in both the left and right toolbars; or a menu dropdown under the menu icon to offer a compact solution for organizing additional actions.

(1.2) Command Palette

• Serves as a search and navigation tool, allowing users to find functions or navigate directly to specific pages.
• A central location where all commands are accessible and can be executed.
• Extensions can utilize the command palette to expose their core functionality to users, enabling seamless access to additional features and functionalities.

2. Ribbon

(2.1) Top icons Where we register main functions or pages of the app

(2.1) Bottom icons Where we register additional & complementary functions/pages of the app

3. Status Tray

(3.1) App status Used to streamline the overall status of the app, including: model downloading, app updating,...

(3.2) Monitor Bar

• Extensions that contribute to app performance, app activity or app information will have their relevant data or function shortcuts displayed here, which provide users with comprehensive understanding related to app performance, activity & additional information.
• It can be a standalone data displayed, shortcut or it can open Bottom Panel
• For ex: system monitor, hardware monitor, app version, social media shortcuts,...

Bottom Panel

• Can be accessed by clicking on items in (3.2) Monitor Bar
• Users can register any relevant information related to any specific Monitor Bar’s functions.

• [Later on] Users should be able to customize their view, choosing between a split layout or a tab layout for their convenience

Content Space

Left sidebar

• Serves as a supplementary space within the content space, offering secondary information architecture tailored to the specific needs of each page.
• Its content varies based on the page's function, such as displaying thread lists for Threads, customization functions for Local API Server or filters for Hub. 

• As a secondary panel, it is optional to have and contingent upon the requirements of the page.
• Users have the option to collapse the sidebar via control panel settings

Right sidebar

• Acts as a dedicated surface for Settings Containers relevant to the current page, presenting all essential Settings Items of that page.
• Here, users can access settings contributed by extensions specific to the page they're on.
• As a secondary panel, it is optional to have and contingent upon the requirements of the page.
• Users have the option to collapse the sidebar via control panel settings

• [Later on] Customization options for these settings are available in both waterfall and tab layouts, providing users with flexibility in organizing and managing their preferences.

Main panel

• A pivotal section within the content space, consistently visible to users due to its significance in presenting the main information, ensures that users can achieve their current tasks or goals.
• Tailored to each page's unique requirements. Input box

• Tools extensions can register features or icons in the input box, such as a microphone icon for speech input tools; or web search under tools selection.

.

By Extensions

Remote API Extensions:

• Engine Settings Registration: Allow Remote API Extensions to register custom inference settings (e.g., context length, temperature) in the Right Panel, ideally in a collapsible section for easy access.
• Dedicated Settings Page: Each API extension can have its own settings page accessible through the main settings menu, providing users with a central place to configure API keys, endpoints, and other extension-specific settings.

Tools Extensions

• Tool Selector in Input Box: Enable a tool selector dropdown within the input box, allowing users to switch between different tools seamlessly (e.g., search, file search, web search).
• Right Panel - Tool-Specific Settings: For tools that require specific settings (e.g., search filters), register collapsible sections in the Right Panel. This keeps the UI organized and accessible while avoiding clutter in the primary workspace.
• Progress, Status & Result in Main Panel:
  • File Search: Display a “searching…” status & results in the main panel.
  • Web Search: Similar to file search, a status & results should show in the main UI. If a Perplexity-like structure is desired, include source references or citations directly in the response area.

[urmauur]

Comment on defined area keys for extensions

• Cases top panel / top bar

We have three columns in the top panel: left, center, and right. The left and right areas are clear to me. Extensions can register a new toolbar in the top panel, defining the position as either left or right. Is the center area, or the Command Palette injectable by extensions? If yes, how UI looks like?

• Cases bottom panel / bottom bar

Let’s say I want to create an extension and choose the UI to display on the bottom panel. Do we allow that? If so, what should the UI look like?

Let’s clarify the areas where extensions can inject UI elements. A clear example for me would be the Thread Right Panel Tab and the Ribbon.