Use Semantic Search for Copilot

shawonibnkamal commented 2 months ago

Description

Describe in detail what your pull request does and why.

Current implementation of copilot uses full text search of postgres. Which is ineffective to find relevant threads when user types a large prompt. So changed the implementation to use embeddings that will be generated for each threads. Then the prompt's embedding will be compared for the closest distance to find the relevant threads.

Also update copilot UI funcitonality. Needed to store history as cache locally. Added a check to allow users to not refer to doclin threads if needed and a check to allow users to refer to currently open code file

Screenshots or screen recordings

Attach screenshots/screen recordings where applicable. Highly recommended for all the changes.
Include before and after images, this will assist reviewers and future readers in understanding the change better.

How to test the change

Outline the steps to set up and validate the change.

[x] Ensure your local has many different discussion threads.
[x] Type a prompt to the copilot, the response should be carved from the existing discussions.
[x] Smoke test create and update threads/replies.

Check this if your PR is ready to merge and a reviewer will merge when approved.

[x] Ready to merge

shawonibnkamal commented 1 month ago

@taprottoy Can you review when get a chance?

taprottoy commented 1 month ago

getting this error when sending prompt

POST /organizations/462c0943-3dba-4333-a437-e1fbf479b28f/projects/1/copilot
Unable to load from local path "/Users/taprottoy/Desktop/prottoy/tp/doclin/api/node_modules/@xenova/transformers/models/Xenova/all-MiniLM-L6-v2/tokenizer.json": "ReferenceError: Headers is not defined"
Unable to load from local path "/Users/taprottoy/Desktop/prottoy/tp/doclin/api/node_modules/@xenova/transformers/models/Xenova/all-MiniLM-L6-v2/tokenizer_config.json": "ReferenceError: Headers is not defined"
Unable to load from local path "/Users/taprottoy/Desktop/prottoy/tp/doclin/api/node_modules/@xenova/transformers/models/Xenova/all-MiniLM-L6-v2/config.json": "ReferenceError: Headers is not defined"
Error in createEmbedding: ReferenceError: Headers is not defined
    at getFile (file:///Users/taprottoy/Desktop/prottoy/tp/doclin/api/node_modules/@xenova/transformers/src/utils/hub.js:185:25)
    at getModelFile (file:///Users/taprottoy/Desktop/prottoy/tp/doclin/api/node_modules/@xenova/transformers/src/utils/hub.js:468:30)
    at async getModelJSON (file:///Users/taprottoy/Desktop/prottoy/tp/doclin/api/node_modules/@xenova/transformers/src/utils/hub.js:572:18)
    at async Promise.all (index 0)
    at async loadTokenizer (file:///Users/taprottoy/Desktop/prottoy/tp/doclin/api/node_modules/@xenova/transformers/src/tokenizers.js:61:18)
    at async Function.from_pretrained (file:///Users/taprottoy/Desktop/prottoy/tp/doclin/api/node_modules/@xenova/transformers/src/tokenizers.js:4459:50)
    at async Promise.all (index 0)
    at async loadItems (file:///Users/taprottoy/Desktop/prottoy/tp/doclin/api/node_modules/@xenova/transformers/src/pipelines.js:3279:5)
    at async pipeline (file:///Users/taprottoy/Desktop/prottoy/tp/doclin/api/node_modules/@xenova/transformers/src/pipelines.js:3219:21)
Uncaught Exception: Headers is not defined
[nodemon] app crashed - waiting for file changes before starting...

taprottoy commented 1 month ago

also throws this error when opening the copilot window for the first time.

copilotSidebar.js:4704 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'scroll')
    at scrollToBottom (copilotSidebar.js:4704:12)

doclin-dev / doclin