Closed dagardner-nv closed 4 months ago
24.03
Source
Using the --stop_after=1024 should cause the source to stop after emitting 1024 records, instead at 1024 this is logged:
--stop_after=1024
Stop limit reached... preparing to halt the source. RSS source exhausted, stopping.
Yet the pipeline continues to execute. It appears that the feed parser continues to endlessly retry failed urls.
python examples/llm/main.py vdb_upload pipeline --stop_after=1024
[Paste the results of print_env.sh here, it will be hidden by default]
No response
Invalid issue, source completes as expected, additional logs are being generated bu the web scraper stage.
Version
24.03
Which installation method(s) does this occur on?
Source
Describe the bug.
Using the
--stop_after=1024
should cause the source to stop after emitting 1024 records, instead at 1024 this is logged:Yet the pipeline continues to execute. It appears that the feed parser continues to endlessly retry failed urls.
Minimum reproducible example
Relevant log output
Click here to see error details
Full env printout
Click here to see environment details
Other/Misc.
No response
Code of Conduct