Migrate AML SDK from v1 to v2

[SimonYansenZhao]

Description

This PR migrate AzureML Python SDK used in recommenders' testing workflow from v1 to v2, because AML Python SDK v1 has the following limitations:

• Support for CLI v1 will end on September 30, 2025
• That we must pin pip=20.1.1 makes the version upgrade difficult.
• Snapshots in v1 seem to be buggy
• Logs are not easy to get

Related Issues

• 2066

• 2098

References

• What is Azure Machine Learning Python SDK v2?
• Should I use v1 or v2?
• Migrate logging from SDK v1 to SDK v2

Checklist:

• [X] I have followed the contribution guidelines and code style for this project.
• [ ] I have added tests covering my contributions.
• [ ] I have updated the documentation accordingly.
• [X] I have signed the commits, e.g. git commit -s -m "your commit message".
• [X] This PR is being made to staging branch AND NOT TO main branch.

[miguelgfierro]

super great job @SimonYansenZhao congrats!