I'm currently exploring the gpt-crawler project and I'm interested in setting up the crawler for multiple web pages, each with a unique configuration. My goal is to have a separate config.ts file for each webpage, allowing for customized settings per page. However, I'm unsure about the best approach to implement this within the project's structure, especially without clear insight into the internal workings of the crawler.
Could you provide guidance or suggestions on how to configure the crawler to use different config.ts files for different web pages? Is there a recommended method to achieve this within the current framework of gpt-crawler, or would this require significant modifications to the codebase?
Any advice or pointers you could provide would be greatly appreciated, as this functionality is crucial for my specific use case. Thank you for your time and for the great work on this project.
Hello gpt-crawler Team,
I'm currently exploring the gpt-crawler project and I'm interested in setting up the crawler for multiple web pages, each with a unique configuration. My goal is to have a separate
config.tsfile for each webpage, allowing for customized settings per page. However, I'm unsure about the best approach to implement this within the project's structure, especially without clear insight into the internal workings of the crawler.Could you provide guidance or suggestions on how to configure the crawler to use different
config.tsfiles for different web pages? Is there a recommended method to achieve this within the current framework of gpt-crawler, or would this require significant modifications to the codebase?Any advice or pointers you could provide would be greatly appreciated, as this functionality is crucial for my specific use case. Thank you for your time and for the great work on this project.
Best regards, Siyao