EPIC: feedback on current code

[Shi-Ho]

General

• [x] rename repo to ZAS/EAK-Copilot?
• parameter question in most functions, especially autocomplete related ones, is a bit confusing (it leads to description like "return a list of questions that match the question"). Rename to input or user_input maybe?
• [x] move all code directories to an src/ folder to have a distinct split between docs and code
• app ports shouldn't be hardcoded, but as default param values
• [x] stored sql procedures and queries sanitarization to avoid code injection [IMPORTANT] -> Use ORM
• regarding fastapi instance, could we put the logging related lines in a lifespan function and parameter?
  • source: https://fastapi.tiangolo.com/advanced/events/
  • we could also have a fastapi subclass that always setup logging at init

Autocomplete

• [x] move all autocompleterelated functions to an autocomplete class (is it possible?)
• more default param values (ex: nb of returned queries, distance threshold)
• would get_fuzzy_match be faster using a panda dataframe? result could be ordered (idk if this is good practice)
  • [x] EDIT: use PostgreSQL native fuzzy match module
  • grouping results (fuzzy+exact) without duplicate could be done within PostgreSQL with this module, but that would weaken the python code flexibility (an sql wrapper class could solve this)
  • if fuzzy matching also return results with distance 0, this necessarily include all exact matching results, so no need to do exact matching and merge results
• use INSERT ... ON DUPLICATE KEY UPDATE in update_or_insert_data, add unique index to query data
  • source: https://stackoverflow.com/a/4205207 and https://stackoverflow.com/a/5786424
• [x] move web_scraper to indexing/app/
• should completion done in a selected language? If so, add language as function parameters (default value '*' as example) for future integrations

Webscraper

• [x] optimize scraper so it gets language, question and answer in a single HTTP request instead of 3

RAG

• [x] same, create a RAG class an move code inside
• more default param values (ex: embedding engine)
• [x] rag.main.doc and autocomplete.main.get_semantic_similarity_match have similar sql queries, is there a way to simplify this? is having a single class that can interact with the db good practice? -> should be solved after refactoring and ORM implementation

Database

• questions column should be flagged UNIQUE (and non empty)

Other code non-related things

• README: section Features followed by a section How it works - The EAK-Copilot currently features: ... is a bit confusing
• PR Form: Requested Reviewers section is also not really useful imo, we can requests reviewers on the right panel when creating a PR.

[K-Schubert]

@Shi-Ho Can you investigate alternative DB solutions for autocomplete and rag. We need something that could ideally perform SQL querying, text (fuzzy) search and vector search.

eg. elasticsearch

[Shi-Ho]

Hi @K-Schubert ! I just found out PostgreSQL has its own fuzzysearch module with Levenshtein distance, which might be quite efficient and more pratical than Elasticsearch that requires setting up another server. More info here: https://www.postgresql.org/docs/current/fuzzystrmatch.html#FUZZYSTRMATCH-LEVENSHTEIN

Also, I've updated the feedback list with some other points

[Shi-Ho]

Some additional information:

• Indexing using Soundex can greatly improve postgresql queries with levenshtein: link
• the module pg_trgm is also interesting and is known to be faster then levenshtein alone: link

[Shi-Ho]

Regarding the current implementation of the Copilot, we have:

• on click on the drop-down list, return autocomplete answer
• on enter or click on the send button, return RAG

However, people will not necessarily click on the autocomplete drop-down list, even if there is very similar questions (similarly, people do not necessarily click on the drop down list results on a Google search even if there's the exact same question), and it might be difficult to ask people to change their habits regarding search engines.

A solution could be, on enter/send button, to return the autocomplete question's answer with the smallest distance from the input, IF that distance if smaller than a very small threshold.

Exemple:

• The database contains "I have a bachelor degree, how much will I earn in the Confederation?"
• The user click on "Enter" after writing "i did a bachelor degre how much will I ear in the confederation".

Currently, because they clicked on "Enter", Copilot will query the RAG. However, the answer could clearly be found in the database. If the Copilot could return the autocomplete instead of query the RAG, this would be cheaper while providing the correct answer.