[Feature] [Connector] Customization of the crawler settings

IBM / data-prep-kit

Open source project for data preparation of LLM application builders

https://ibm.github.io/data-prep-kit/

Apache License 2.0

307 stars 134 forks source link

[Feature] [Connector] Customization of the crawler settings #737

Closed hmtbr closed 3 weeks ago

hmtbr commented 4 weeks ago

Search before asking

[X] I searched the issues and found no similar issues.

Component

Other

Feature

The data-prep-connector doesn't allow customization of the crawler settings like request concurrency, delay and timeouts. We need to update the library so that the user can specify their own settings.

Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!