Open B4nan opened 7 months ago
Hello from the crawlee team!
Just a small suggestion, I was taking a peek at the code and saw you do this to create a data bundle at the end.
https://github.com/BuilderIO/gpt-crawler/blob/27b65d36d42660ad925ec36c399f717b8f359726/src/core.ts#L115-L129
We recently added a new helper that does exactly the same:
https://crawlee.dev/api/basic-crawler/class/BasicCrawler#exportData
So you could replace the whole function with a simple crawler.exportData(config.outputFileName) call, and it will support both JSON and CSV automatically (based on the file extension).
crawler.exportData(config.outputFileName)
Hello from the crawlee team!
Just a small suggestion, I was taking a peek at the code and saw you do this to create a data bundle at the end.
https://github.com/BuilderIO/gpt-crawler/blob/27b65d36d42660ad925ec36c399f717b8f359726/src/core.ts#L115-L129
We recently added a new helper that does exactly the same:
https://crawlee.dev/api/basic-crawler/class/BasicCrawler#exportData
So you could replace the whole function with a simple
crawler.exportData(config.outputFileName)call, and it will support both JSON and CSV automatically (based on the file extension).