NOTES ABOUT CODE KHOJ

[stevennt]

https://docs.khoj.dev/get-started/setup

Where is the FastAPI initiated / called ?

What are the endpoints I can use to call from outside?

I want to use its capability as backend for my other tasks, such as filling out RFPs.

[stevennt]

Indexing files: src/khoj/routers/indexer.py

[stevennt]

Initial Data: how to code them down instead of manually doing it on the frontend? http://localhost:42110/server/admin/

[stevennt]

Accepted files: src/khoj/interface/web/chat.html

[stevennt]

API to answer chat: // Generate backend API URL to execute query let url = /api/chat?q=${encodeURIComponent(query)}&n=${resultsCount}&client=web&stream=true&conversation_id=${conversationID}&region=${region}&city=${city}&country=${countryName}&timezone=${timezone};

        // Call specified ABN API
        let response = await fetch(url);
        let rawResponse = "";
        let references = null;

[stevennt]

Maybe loading the indication here:

[stevennt]

[stevennt]

Database: src/khoj/database/models/init.py

[stevennt]

Init: maybe change here:

src/khoj/utils/initialization.py

[stevennt]

[stevennt]

how / when are the models downloaded?

[stevennt]

Seems that it will download from HuggingFace at runtime.

[stevennt]

Oh they have reranking

[stevennt]

Compute Embeddings, Load Pre-computed embeddings: src/khoj/search_type/text_search.py

[stevennt]

src/khoj/processor/conversation/prompts.py many prompts

[stevennt]

Go to the OpenAI settings in the server admin settings to add an OpenAI processor conversation config. This is where you set your API key and server API base URL. The API base URL is optional - it's only relevant if you're using another OpenAI-compatible proxy server. Go over to configure your chat model options. Set the chat-model field to a supported chat model1 of your choice. For example, you can specify gpt-4-turbo-preview if you're using OpenAI. Make sure to set the model-type field to OpenAI. The tokenizer and max-prompt-size fields are optional. Set them only if you're sure of the tokenizer or token limit for the model you're using. Contact us if you're unsure what to do here. Configure Offline Chat No need to setup a conversation processor config! Go over to configure your chat model options. Set the chat-model field to a supported chat model1 of your choice. For example, we recommend NousResearch/Hermes-2-Pro-Mistral-7B-GGUF, but any gguf model on huggingface should work. Make sure to set the model-type to Offline. Do not set openai config. The tokenizer and max-prompt-size fields are optional. Set them only when using a non-standard model (i.e not mistral, gpt or llama2 model) when you know the token limit.

[stevennt]

[stevennt]

Successfully configure Khoj with OpenAI:

[stevennt]

src/khoj/database/models/init.py

[stevennt]

src/khoj/migrations/migrate_processor_config_openai.py

[stevennt]

The URL should be without /chat, because Khoj appends that automatically. If I add, it will be duplicate.

[stevennt]

BadRequestError: Error code:
myaiabnkhoj-server-1 | 400 - {'error': {'message':
myaiabnkhoj-server-1 | 'response_format` does not
myaiabnkhoj-server-1 | support streaming', 'type':
myaiabnkhoj-server-1 | 'invalid_request_error'}}

[stevennt]

PROMPTS: src/khoj/processor/conversation/prompts.py

[stevennt]

[stevennt]

src/khoj/configure.py

[stevennt]

https://docs.khoj.dev/get-started/setup/

The tokenizer and max-prompt-size fields are optional. Set them only if you're sure of the tokenizer or token limit for the model you're using. Contact us if you're unsure what to do here.

[stevennt]

src/khoj/processor/conversation/utils.py

[stevennt]

def truncate_messages( messages: list[ChatMessage], max_prompt_size, model_name: str, loaded_model: Optional[Llama] = None, tokenizer_name=None, ) -> list[ChatMessage]: """Truncate messages to fit within max prompt size supported by model"""

default_tokenizer = "hf-internal-testing/llama-tokenizer"

try:
    if loaded_model:
        encoder = loaded_model.tokenizer()
    elif model_name.startswith("gpt-"):
        encoder = tiktoken.encoding_for_model(model_name)
    elif tokenizer_name:
        if tokenizer_name in state.pretrained_tokenizers:
            encoder = state.pretrained_tokenizers[tokenizer_name]
        else:
            encoder = AutoTokenizer.from_pretrained(tokenizer_name)
            state.pretrained_tokenizers[tokenizer_name] = encoder
    else:
        encoder = download_model(model_name).tokenizer()
except:
    if default_tokenizer in state.pretrained_tokenizers:
        encoder = state.pretrained_tokenizers[default_tokenizer]
    else:
        encoder = AutoTokenizer.from_pretrained(default_tokenizer)
        state.pretrained_tokenizers[default_tokenizer] = encoder
    logger.warning(
        f"Fallback to default chat model tokenizer: {tokenizer_name}.\nConfigure tokenizer for unsupported model: {model_name} in Khoj settings to improve context stuffing."
    )

[stevennt]

Lets try this: google-bert/bert-base-uncased https://huggingface.co/docs/transformers/en/main_classes/tokenizer

[stevennt]

openai-community/gpt2

[stevennt]

Oh, in the code: default_tokenizer = "hf-internal-testing/llama-tokenizer"

[stevennt]

http://localhost:42110/server/admin/database/agent/1/change/

[stevennt]

Upload Files: src/khoj/interface/web/chat.html

[stevennt]

All the things (uploads, etc.) are implemented in API, so I can just play with the APIs

[stevennt]

src/khoj/routers/api_chat.py

[stevennt]

Sync/index data: Simply edit this config file and let Khoj Desktop do the job.

{ "files": [ { "path": "/home/thanhson/Downloads/RFP#2024-Amgen-01 Biding App Upgrade.pdf" } ], "folders": [], "khojToken": "kk-yHlnpZ4zKsw-ocgn9_WxUPRkgl4Fa3cECmNACl4XmVA", "hostURL": "https://app.khoj.dev", "lastSync": [] } ~

[stevennt]

The backup seems to work. But where does it store?

[stevennt]

[stevennt]

[stevennt]

@indexer. @auth_router. @web_client. @subscription_router. @notion_router. @api_chat. @api_agents.

[stevennt]

maybe change this has_documents to initialize with initial documents:

has_documents

[stevennt]

Embeddings: src/khoj/processor/embeddings.py

Text Search: src/khoj/search_type/text_search.py

[stevennt]

[stevennt]

I created my own embeddings and search at ABNScripts

[stevennt]

Prompts, nice: /home/thanhson/Workspace/myai.abn.khoj/src/khoj/processor/conversation/prompts.py from langchain.prompts import PromptTemplate

Personality

--

personality = PromptTemplate.from_template( """ You are ABNCopilot, a smart, inquisitive and helpful personal assistant. Use your general knowledge and past conversation with the user as context to inform your responses. You were created by AbnAsia.org. with the following capabilities:

• You CAN REMEMBER ALL NOTES and PERSONAL INFORMATION FOREVER that the user ever shares with you.
• Users can share files and other information with you using the Khoj Desktop, Obsidian or Emacs app. They can also drag and drop their files into the chat window.
• You CAN generate images, look-up real-time information from the internet, set reminders and answer questions based on the user's notes.
• Say "I don't know" or "I don't understand" if you don't know what to say or if you don't know the answer to a question.
• Make sure to use the specific LaTeX math mode delimiters for your response. LaTex math mode specific delimiters as following
  • inline math mode : \\( and \\)
  • display math mode: insert linebreak after opening $$, \\[ and before closing $$, \\]
• Ask crisp follow-up questions to get additional context, when the answer cannot be inferred from the provided notes or past conversations.
• Sometimes the user will share personal information that needs to be remembered, like an account ID or a residential address. These can be acknowledged with a simple "Got it" or "Okay".
• Provide inline references to quotes from the user's notes or any web pages you refer to in your responses in markdown format. For example, "The farmer had ten sheep. 1". ALWAYS CITE YOUR SOURCES AND PROVIDE REFERENCES. Add them inline to directly support your claim.

Note: More information about you, the company or ABN apps for download can be found at https://abnasia.org. Today is {current_date} in UTC. """.strip() )

custom_personality = PromptTemplate.from_template( """ You are {name}, an Ai agent from ABN Asia. Use your general knowledge and past conversation with the user as context to inform your responses.