Azure / MachineLearningNotebooks

Python notebooks with ML and deep learning examples with Azure Machine Learning Python SDK | Microsoft
https://docs.microsoft.com/azure/machine-learning/service/
MIT License
4.1k stars 2.52k forks source link

Raise exception when provisioning fails #1524

Open JackCaster opened 3 years ago

JackCaster commented 3 years ago

Currently, no error is thrown when the schedule fails to provision. Only a message is printed in the console 'Provisioning failed, please try again or contact support.':

# schedule.py
def _wait_for_provisioning(self, timeout_seconds):
        """
        Wait for the completion of provisioning for this schedule.

        Returns the status after the wait.

        :param timeout_seconds: Number of seconds to wait before timing out.
        :type timeout_seconds: int

        :return: The final status.
        :rtype: str
        """
        status = self._schedule_provider.get_schedule_provisioning_status(self.id)
        time_run = 0
        sleep_period = 5
        while status == 'Provisioning':
            if time_run + sleep_period > timeout_seconds:
                print('Timed out of waiting, status:%s.' % status, flush=True)
                break
            time_run += sleep_period
            time.sleep(sleep_period)
            status = self._schedule_provider.get_schedule_provisioning_status(self.id)
            print('Provisioning status:', status)
        if status == "Failed":
            print('Provisioning failed, please try again or contact support.')
        return status

If the provisioning fails, it is easy for the user to miss that. Also, there is no way the process can be safeguarded in production environments (e.g., stop the process if the schedule has not being properly provisioned). On top of this, when the provisioning fails, the schedule will still appear as Active (see also https://github.com/MicrosoftDocs/azure-docs/issues/77089).


Document Details

Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

shbijlan commented 3 years ago

@JackCaster thank you for the feedback. I have reported this issue to engineering and will update after investigation here.