(EAI-252, EAI-253) Ingest & Chunk Code Files

[nlarew]

Jira:

• (EAI-252) [Ingest] Code as text files
• (EAI-253) [Ingest] Language-aware code chunking

Changes

• Adds CodeOnGithubTextDataSource to ingest code from a repo in txt format
• Adds a partial implementation of MongoDB Developer repos in mongodbDeveloperCodeRepos
• Adds language-aware chunking for everything supported by langchain

Notes

• The repos in mongodbDeveloperCodeRepos are mainly just an example for now. We don't actually include them in our list of sources to ingest.
• There is a lot of config if we want to include all these repos. We should instead consider a pull-based approach e.g. https://jira.mongodb.org/browse/EAI-270

[mongodben]

The repos in mongodbDeveloperCodeRepos are mainly just an example for now. We don't actually include them in our list of sources to ingest.

what are you thinking for next steps here @nlarew?

[nlarew]

The repos in mongodbDeveloperCodeRepos are mainly just an example for now. We don't actually include them in our list of sources to ingest.

what are you thinking for next steps here @nlarew?

@mongodben Once the broader implementation here is LGTM'd I'd like to run an A/B test with our quality evaluation to see if/how the raw code chunks help