Make GPT Crawler a CLI

[marcelovicentegc]

1k+ changes from package-lock.json

Essentially:

• Makes GPT Crawler a CLI.
• Splits the code between:
  • core.ts: the core methods live here
  • main.ts leverages the methods from core.ts to run the service with config file
  • cli.ts: leverages the methods from core.ts to run the service as a CLI

What are the next steps?

• [ ] To setup this package under @BuilderIO on NPM (or any other registry). We would basically just need a NPM_TOKEN with write access setup on this repository. Can you help @steve8708 :hugs:?
• [x] Are you ok on using GItHub workflows to ship the CLI?
• [x] Are you ok on using semantic-release to take care of versioning this package?

https://github.com/BuilderIO/gpt-crawler/pull/56 addresses the aforementioned changes ☝️

I can address all these on different PRs but need your feedback. Let's keep in touch :wink: .

Other changes that are not directly related to the CLI and that I can revert back to what it was or open other PRs:

• Adds preinstall script for automatic playwright setup
• Improves the Config interface jsDocs with @example and @default decorators
• Adds a table of contents to the docs and separates CLI concerning content from development content

Resolves #37

[databill86]

Thank you for this pull request. I was wondering if it's possible to expose the service as an API. It would be a lot easier and simpler to run it locally, without the need to publish the gpt crawler. I'm no expert in js, I tried to implement an express js server with the help of chatgpt, but I had a lot of exceptions and errors, so I gave up ^^

[marcelovicentegc]

Thank you for this pull request. I was wondering if it's possible to expose the service as an API. It would be a lot easier and simpler to run it locally, without the need to publish the gpt crawler. I'm no expert in js, I tried to implement an express js server with the help of chatgpt, but I had a lot of exceptions and errors, so I gave up ^^

Hey @databill86! I've commented about your feature request on #43 😉, but TLDR; I like the idea!

[steve8708]

This is awesome @marcelovicentegc! Also just realized that you work at VTEX, awesome, I know Mariano and Geraldo (they are amazing) and I went to VTEX Day last year!

To answer your questions: lets definitely setup the workflow process through github actions. I can happily add an NPM_TOKEN to the secrets there. And semantic release is perfect.

Also, I recall you saying this was WIP, is it ready for review on your side or worth still waiting on?

[marcelovicentegc]

How cool, @steve8708! This is awesome! I was at VTEX Day this year ❤️. It's such a great event.

I will set up semantic-release + GitHub actions on another PR, since this one is already large, ok? 🤗

And finally, the PR is rebased and good for review 👍

[github-actions[bot]]

:tada: This PR is included in version 1.0.0 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket: