MicrosoftLearning / mslearn-knowledge-mining

Lab files for Azure AI Knowledge Mining modules
https://microsoftlearning.github.io/mslearn-knowledge-mining/
MIT License
65 stars 126 forks source link

__init__.py: improve error handling and enhance async processing #28

Closed thor4 closed 1 month ago

thor4 commented 5 months ago

I experienced many errors when trying to run the indexer due to async issues. It was difficult at first to discern what was driving the failures. So, I modified the async approach and improved error handling.

Key changes:

  1. Improved error handling:

    • Used specific exceptions like KeyError and RequestException for more granular error handling.
    • Logged errors using logging.error() for better visibility and troubleshooting.
    • Raised exceptions using raise_for_status() for 4xx and 5xx status codes from the Text Analytics API.
    • Handled job failure by raising a RuntimeError if the job status is 'failed'.
  2. Enhanced asynchronous processing:

    • Implemented a retry mechanism with exponential backoff to handle cases where the Text Analytics job takes longer to complete.
    • Added a maximum number of retries (max_retries) and a retry delay (retry_delay) that doubles with each retry attempt.
    • Raised a TimeoutError if the job doesn't complete within the maximum number of retries.
ivorb commented 1 month ago

There are some additional changes we want to include here, we appreciate these updates and improvements, but since this is a zip I don't want to merge as is. Thanks for your contributions! Update with some of these changes will be coming shortly