feat: create crawler api server

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

https://www.builder.io/blog/custom-gpt

ISC License

18.59k stars 1.97k forks source link

feat: create crawler api server #52

Closed adityak74 closed 9 months ago

adityak74 commented 10 months ago

This PR creates an express server that exposes /crawl endpoint that accepts the config as a post-request body and responds to the crawled json.

Swagger API Documentation at: /api-docs

Doesn't support concurrency yet. - #120

[x] Create a basic structure for the server
[x] Modularize port and host for deployments
[x] Add validations for config parsing and conversion.
[x] Swagger UI for API documentation

Fixes: #43

BikeF commented 10 months ago

I did not run your changes, but I am concerned that this API does not support concurrent calls. He cannot be used simultaneously

databill86 commented 9 months ago

Any news on this feature ? :)

adityak74 commented 9 months ago

@databill86 I am working on the docker file, facing some small issues, I should get it in a couple of days.

adityak74 commented 9 months ago

@databill86 - This PR is ready to be reviewed. I will attempt to Dockerize in another PR. This should get us started with the API running locally.

steve8708 commented 9 months ago

looks great, thanks for the contribution @adityak74! would be great if you could send a followup PR updating the README about how to use this as an API as well, so new users know what is possible and how to use these updates

github-actions[bot] commented 9 months ago

:tada: This PR is included in version 1.2.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

adityak74 commented 9 months ago

@steve8708 - Thanks and sure I will create README PR soon.