(See changelog here)
Controls including vocal transcription:
NB: API fees should not be confused with the ChatGPT Plus subscription; they are strictly separate. You do not need the ChatGPT plus subscription to use Live AI Assistant.
To use Claude models, provide your Anthropic API key (by copying/pasting an existing key or generating a new one via this link).
To use other existing models, you can provide an OpenRouter API Key or a Groq API Key. You can define OpenRouter as your default model provider or use it as a complement to direct access to OpenAI and Anthropic API. Using Groq, you can also replace default Whisper model by whisper-large-v3
model!
To use local models with Ollama, see 'Use Ollama to run local models' section below.
Just press the microphone button and provide vocal instructions, or place the cursor focus in a block where you have written your prompt, then click on the AI completion button (OpenAI Logo). That's all !
You can easily use structured prompts by selecting multiple blocks (including images with models supporting image recognition). Create your own Roam template to have a set of ready-to-use advanced prompts !
You can easily add context to your prompt: by pressing Shift
while clicking on the AI completion button, all the content in the sidebar will be sent as context (for example, ask to resume some content provided in the context). See 'AI Assistant' section below for more details and possibilities about context.
You can easily compare AI models responses: right click on 'Generate a response again' button ⟳
appearing on the right of the AI response and choose another model. The new response will be inserted just above the first one.
You can easily continue any conversation with an AI Assistant:
AI Assistant (<model>):
). It is not necessary to insert your role (such as "Me:") but more convenient in terms of readability.Previous messages (including children blocks) will be automatically taken into account. About the context:
Spacebar
Escape
or Backspace
T or Enter
E
C
P
Trigger controls concerning vocal notes:
Keyboard-only (no vocal) interactions with the AI assistantAI features and other commands:
A SmartBlock command is also provided: <%SPEECHTOROAM%>
, see the example of SmartBlock at the end of this doc.
⚠️ Currently, voice recording isn't possible on either the MacOS desktop app or the Mobile app : microphone is not yet supported, so vocal notes transcription can't be achieved. But all commands relying only on text (like AI completion or post-processing) are available. The extensions works properly on all browsers (desktop and mobile, MacOs, iOS, Windows or Android) and on Windows desktop app.
whisper-large-v3
model.A large number of source languages are supported, but the target language is currently limited to English. This limitation can be easily overcome through post-processing using a GPT model, as it only requires asking it to translate into almost any language.
ask any question (by speaking or writing, focusing on a block or selecting one or more blocks as prompt), rephrasing, completion, translation... The more precise your instructions are (especially with examples and context), the more accurate and satisfying the responses will be.
assistant response is inserted as child of prompt block (by default, the answer is splitted in as many blocks as it contains paragraphs. There is an option to always limit the response to a block.). The assistant name is inserted before its response and include by default the AI model name. The template can be changed in the settings.
🆕 in v.4: subtle buttons appear to the right of the first block of the assistant's response. They allow to stop the streamed response, to generate again the response (possibly with a different AI model by selecting it from the context menu accessed by right-clicking) and to copy the response to clipboard. These are temporary buttons that disappear when changing the page or if the block is collapsed.
you can easily provide context to your prompt, to process the information contained in this context (e.g. to summarize it or have a conversation with your own notes 🚀) or to provide resources to the AI for the required text generation:
Command/Control
: use linked references (backlinks) of the current page or, if the page is a DNP or the daily notes log (journal), the previous DNP. Important limitation for linked refs: currently, the applied filters (including or excluding some reference) are not taken into account.Alt
: use the whole main page (more precisely: its current zoomed view) as contextShift
: use all the content in the right sidebar (pages, blocks, linked references)⚠️ Attention, be aware that using linked refs or DNPs as context can easily reach the model's maximum token limit and quickly become costly. I highly recommend combining multiple questions or instructions into a single request, as the entire context needs to be sent with each request. That can be facilitated by the template-based prompt functionality presented below. To reduce costs and increase the usable context window, the following options can be configured:
#private
)for a more "surgical" defined context, simply select the required blocks. Using the native multi-select blocks feature to choose multiple separate blocks. The selection can be done before recording or updated just before sending the transcription to the GPT model. The block content of the initial focused block provide the initial context if no selection is made subsequently. As for simple transcription, the focused block right before completion is the target where your prompt and the answer will be inserted.
the context provided by selected blocks is handled with the following instructions (you can add your own instructions, see settings):
"Below is the context of your response, it can consist of data to rely on, a conversation to be continued, or other instructions, depending on the user's prompt. The user can refer to it as 'these blocks' or 'the selected blocks' among other possibilities. The ((9-characters code)) within double parentheses preceding each piece of content (or block) in the context is its id in the database and is called 'block reference'. In your response, you can refer to an existing block reference (and only existing one) if needed, using the syntax [](((9-characters code))) to refer to it as a note or citation. Example: *>). If you need reproduce the entire and exact content of a block in your response, only put its ((9-characters code)) instead, but it's preferable to prioritize a citation with [](((9-char code))) syntax most of the time. Expressions within double brackets such as [[my page]] or preceded by a '#' like #tag or #[[my tag]] should be reused with the exact same syntax as in the source text, keeping the original double brackets and/or hashtag: e.g. [[my page]], #tag, #[[my tag]]."
on mobile (selection being not possible), place the cursor in a block to use its content as context, or enable the option to use the whole current view as context (unless you place the cursor in a blank block). You can also insert in a block a command to define the context, see below the section "Keyboard & text only AI completion and post-processing".
you can customize the AI assistants's "character", its default character is defined this way (you can add your own definition of its character, see settings):
"You are a smart, rigorous and concise assistant. You always respond in the same language as the user's prompt unless specified otherwise in the prompt itself."
model by default is currently gpt-4o-mini
(pointing to the latest model version). You can change the default model or choose any other model (among GPT 4 and Claude models) for each request by right-clicking on the completion button (just like for Post-processing button and Generate again button)
you can try other chat completion model, or your own fine-tuned models (only OpenAI chat completion).
/Template
command, but it's not required);;
command) as a child of a given block.OpenRouter is an unified API routing requests to wide range of models. The benefit is having a single account to access to most of existing and up-to-date models. You pay as you go: after purchasing credit (you can test without credit), your credit is debited on each request. OpenRouter also offers a continuously updated ranking of the most popular models.
In the settings, provide the list of IDs of the models you want to use in LiveAI. They will appear in the context menu in a dedicated section or replace the native models if you check the corresponding option. The first model in your list can be selected as your default model.
By default, logging of your inputs & outputs in OpenRouter's settings is enabled, you can disable it from your OpenRouter account.
Ollama allows you to run local models like Llama3, so all your data shared with the AI assistant is processed entirely locally and is not sent to a third party like OpenAI or Anthropic. (Please note: a local model is typically slower than a remote model and requires a machine with a lot of RAM. E.g a 7B model may require 7GB of RAM to work properly)
Install Ollama, install a model (ex. ollama run llama3
), add the model name in the settings above (e.g. llama3
), and follow the instructions below:
To use Ollama in Roam, you have also to set OLLAMA_ORIGINS environment variable to https://roamresearch.com
(by default, Ollama CORS is restricted to local origins). See Ollama documentation here or proceed this way, according to your operating system:
~/.zshrc
file and add export OLLAMA_ORIGINS="https://roamresearch.com"
command. The environment variable will be set at OS startup or when opening the zsh terminal. (To edit a file, open the terminal and run a text editor, e.g. nano ~/.zshrc
. Save changes with Ctrl+x, Y and Enter). Close and open again the terminal. (You can also set this variable temporarily using the command launchctl setenv OLLAMA_ORIGINS "https://roamresearch.com"
and restart the terminal)⚠️ In my experience, MacOS Ollama.app doesn't take into account OLLAMA_ORIGINS variable change. After Ollama installation, Ollapa.app will be loaded in the background. You need to close it (using, e.g., the activity monitor), then launch "ollama serve" from the terminal. It may also be necessary to disable the automatic startup of Ollama.app when your OS starts by going to System Preferences > General > Startup > Open at login: select Ollama.app and click on the minus sign (-).
https://roamresearch.com
systemctl edit ollama.service
to open the service file in an editor.[Service]
section, add: Environment="OLLAMA_ORIGINS=https://roamresearch.com"
systemctl daemon-reload
and systemctl restart ollama
commandsYou can also use AI assistant feature without vocal note, just using text content of some blocks in your graph and the dedicated command in command palette (see above).
((context: linkedRefs, sidebar, mainPage, logPages))
.
logPages
means daily log. You can specify the number of DNP to take process (a default limit is fixed to 7 in the settings), this way: ((context: logPages(31)))
.((template: ((block-reference))))
.You can insert <%SPEECHTOROAM%>
command in your SmartBlocks template (using the corresponding extension) to start recording a vocal note in a specific context. You can for example create a very simple SmartBlock and call it with a button:
- #SmartBlock Speech-to-Roam
- <%SPEECHTOROAM%><%CURSOR%>
The SmartBlock button will be {{🎙️:SmartBlock:Speech-to-Roam}}
(can be used once), or to have a permanent button in a given block, and automatically insert the transcription in the children blocks: {{🎙️:SmartBlock:Speech-to-Roam:RemoveButton=false}}
Moderate but regular use should only cost a few tens of cents per month (costs may increase if you use GPT-4 (default is GPT-3.5), think to set a maximum monthly limit). You can check the detailed daily cost of your usage of Whisper and other OpenAI models here, update is almost instantaneous.
OpenAI Whisper API pricing:
$0.006/minute
(rounded to the nearest second)To give you an idea, using Whisper for 10 minutes a day for a month equates to 1.80 $
OpenAI GPT API pricing:
The prices are for 1000 tokens. For comparison, this documentation is equivalent to about 3500 tokens (2000 words).
$0.00015
/ 1K tokens$0.0006
/ 1K tokens$0.0005
/ 1K tokens$0.0015
/ 1K tokens$0.005
/ 1K tokens$0.015
/ 1K tokens$0.01
/ 1K tokens$0.03
/ 1K tokensSee updated OpenAI API pricing here.
Claude API pricing:
$0.00025
/ 1K tokens$0.00125
/ 1K tokens$0.003
/ 1K tokens$0.015
/ 1K tokens$0.015
/ 1K tokens$0.075
/ 1K tokensSee updated Anthropic Claude API pricing here.
This extension represents a significant amount of work. If you want to encourage me to develop further and enhance it, you can buy me a coffee ☕ here. Thanks in advance for your support! 🙏
Please report any issue here.