jimmc414 / 1filellm

Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion

MIT License

432 stars 41 forks source link

Update onefilellm.py #14

Closed dickiesanders closed 3 months ago

dickiesanders commented 4 months ago

Adding chunking for large text to prevent encoding errors in tiktoken.

13

before:

Screenshot 2024-05-24 at 10 59 48 AM

after:

Screenshot 2024-05-24 at 10 59 56 AM