search

brianpetro / obsidian-smart-connections

Chat with your notes & see links to related content with AI embeddings. Use local models or 100+ via APIs like Claude, Gemini, ChatGPT & Llama 3
https://smartconnections.app
GNU General Public License v3.0
2.84k stars 185 forks source link

Stuck in embedding construction #774

Closed MarkusDegen closed 2 months ago

MarkusDegen commented 2 months ago

Tried 3 times now. Deleted smart connections folder too, restart, deactivate plugin etc

When generating the embeddings with OpenAI text embedding small, the process keeps stuck always at 13550/19637

I did not see any log files or so to give more details. I have no idea at which files he stopped

image

MarkusDegen commented 2 months ago

bge-micro-v2 did not stop at 13550

MarkusDegen commented 2 months ago

Found the issue, i used <|endoftext|> in one of the notes. This is GPTs end of text token. I think ticket can be closed

Uncaught (in promise) Error: The text contains a special token that is not allowed: <|endoftext|> at _Tiktoken.encode (plugin:smart-connections:12332:15) at SmartEmbedOpenAIAdapter.count_tokens (plugin:smart-connections:12413:21) at SmartEmbedOpenAIAdapter.prepare_embed_input (plugin:smart-connections:12445:34) at eval (plugin:smart-connections:12427:70) at Array.map () at SmartEmbedOpenAIAdapter.embed_batch (plugin:smart-connections:12427:51) at _SmartEmbedModel.embed_batch (plugin:smart-connections:11101:31) at SmartBlocks.process_embed_queue (plugin:smart-connections:9543:30) at async SmartSources.process_import_queue (plugin:smart-connections:10640:5) at async SmartSources.process_load_queue (plugin:smart-connections:10624:5) at async _SmartEnv.load_collections (plugin:smart-connections:5980:9) at async _SmartEnv.init_collections (plugin:smart-connections:5973:42) at async _SmartEnv.init (plugin:smart-connections:5961:5) at async _SmartEnv.create (plugin:smart-connections:5874:7) at async SmartConnectionsPlugin.load_env (plugin:smart-connections:12676:5) at async SmartConnectionsPlugin.initialize (plugin:smart-connections:12657:5)