BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.14k stars 1.88k forks source link

Custom JSON name and latest dependencies #121

Open miguelgargallo opened 6 months ago

miguelgargallo commented 6 months ago

This pull request introduces a refined structure for the defaultConfig in our TypeScript configuration, enhancing its readability and maintainability. Changes include the decomposition of the URL and file naming into constituent components, allowing for more flexible customization. Additionally, all project libraries and dependencies have been updated to their latest versions, ensuring compatibility and incorporating the latest features and security improvements.

Key Changes:

  1. Refactored defaultConfig for improved clarity:

    • Introduced variables for protocol, domain, top-level domain (TLD), and URL segments.
    • Concatenated these variables to construct the url and match properties.
    • Dynamically set the outputFileName using the domain name.
  2. Updated all libraries and dependencies to their most recent versions.

Benefits:

Your review and approval of these changes would be greatly appreciated.