Add a hook to validate chunks before sending them to be embedded. This can be used for deduplication.
Verification
Can see the chunks being printed out inside the mocked validation endpoint, then later the embeddings are received when they are sent back to the same API as the webhook_url
Can see that the job succeeds:
Timeout
Can see that the timeout will first trigger in the mock validator API:
You can see in the testing client that the upload fails:
Also the timeout exception being thrown is confirmed in the logs:
What
Add a hook to validate chunks before sending them to be embedded. This can be used for deduplication.
Verification
Can see the chunks being printed out inside the mocked validation endpoint, then later the embeddings are received when they are sent back to the same API as the
webhook_urlCan see that the job succeeds:
Timeout
Can see that the timeout will first trigger in the mock validator API:
You can see in the testing client that the upload fails:
Also the timeout exception being thrown is confirmed in the logs: