Improve source-code indexing

[anbraten]

Goals

• add quotes linking to actual code parts
• improve understanding based on files (a file normally contains code that belongs to each other)
• add more metadata to documents (maybe function names, class names, filenames)

[anbraten]

Some links

• https://python.langchain.com/docs/use_cases/code_understanding#loading
• https://python.langchain.com/docs/integrations/document_loaders/source_code
• https://api.python.langchain.com/en/latest/document_loaders/langchain.document_loaders.parsers.language.language_parser.LanguageParser.html
• https://github.com/github/CodeSearchNet
• metadata can be passed to documents like this: Document(page_content=f.read(), metadata={"source": filename, "lines"...})
• https://github.com/JinghaoZhao/GPT-Code-Learner/

[anbraten]

https://github.com/tree-sitter/tree-sitter looks nice for understanding code parts