AzureML v2 sdk - Retrieving a job fails

Package Name: azure-ai-ml
Package Version: 1.16.0
Operating System: Windows / MacOS
Python Version: 3.11.8

Describe the bug

We have the following code in our system:

       mlflow.set_experiment(experiment_name)
       with mlflow.start_run() as run:
               run_id = run.info.run_id
               job = self.__ml_client.jobs.get(run_id)

The last line fails and produces the following error:

    job = self.__ml_client.jobs.get(run_id)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/azure/ai/ml/_telemetry/activity.py", line 372, in wrapper
    return_value = f(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/azure/ai/ml/operations/_job_operations.py", line 351, in get
    job_object = self._get_job(name)
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/azure/ai/ml/operations/_job_operations.py", line 1030, in _get_job
    return self._operation_2023_02_preview.get(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/azure/ai/ml/_restclient/v2023_04_01_preview/operations/_jobs_operations.py", line 614, in get
    raise HttpResponseError(response=response, model=error, error_format=ARMErrorFormat)
azure.core.exceptions.HttpResponseError: (UserError) A job was found, but it is not supported in this API version and cannot be accessed.
Code: UserError
Message: A job was found, but it is not supported in this API version and cannot be accessed.
Additional Information:Type: ComponentName
Info: {
    "value": "managementfrontend"
}Type: Correlation
Info: {
    "value": {
        "operation": "---",
        "request": "---"
    }
}Type: Environment
Info: {
    "value": "---"
}Type: Location
Info: {
    "value": "--"
}Type: Time
Info: {
    "value": "2024-05-28T10:41:25.0211946+00:00"
}Type: InnerError
Info: {
    "value": {
        "code": "NotSupported",
        "innerError": {
            "code": "JobNotSupported",
            "innerError": null
        }
    }
}Type: MessageFormat
Info: {
    "value": "A job was found, but it is not supported in this API version and cannot be accessed."
}Type: MessageParameters
Info: {
    "value": {
        "0": "---"
    }
}

To Reproduce Steps to reproduce the behavior: Just by running the code provided on bug description

Expected behavior A Job object is retrieved

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

Azure / azure-sdk-for-python

AzureML v2 sdk - Retrieving a job fails #35788