search

jimmc414 / 1filellm

Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion
MIT License
265 stars 23 forks source link

Using 1filellm open large git repository and generate prompts for LLMs #15

Open LZING opened 4 days ago

LZING commented 4 days ago

Hi, jimmc414. Thank you for a very nice application.

I'm running into a problem right now when dealing with large code repositories. When I'm dealing with small code repositories, 1filellm works great. But when I'm dealing with large repositories, I have a token overflow problem when interacting with LLM.

So how should we deal with large code repositories? Sending only part of the source code will affect the context. Now it seems that only Gemini 1.5pro can handle about 200m tokens, which is the upper limit.

Do you have any good suggestions?

jimmc414 commented 2 days ago

Hi, LZING thank you! do you have an example repo URL I can use for testing?

LZING commented 2 days ago

Hi, LZING thank you! do you have an example repo URL I can use for testing?

Thx jimmc414, we are using the framework of this project. This project is quite large, and all ts and js add up to nearly 400M tokens. I think it should be divided into modules or folders to process. The repo address is: https://github.com/backstage/backstage/tree/master