Knowledge - Scraping of files reports failure relating to "signal: killed"

[sangee2004]

Steps to reproduce the problem:

1. Create an agent with knowledge files added using "Website" option for https://ranchermanager.docs.rancher.com/
2. There are about 6070 files that got scraped last time I had tested this - https://github.com/otto8-ai/otto8/issues/135

I see the following error reported after scrapping about ~3000 files.

Expected Behavior: Scraping of all 6000+ files should be successful.

Related slack thread - https://acorn-io.slack.com/archives/C07FZ46QA2J/p1730826800094579

[cjellick]

in this case, the scrape process was killed but it does not seem related to a server restart.

[sangee2004]

I see the same error when trying to sync and ingest all files using onedrive link to "corp docs". I picked Automatically add to knowledge as Ingestion policy when adding this link.

When this issue is hit , we do not attempt to automatically sync/ingest the remaining files.