Feature/monitoring

[srastatter]

[1.2.9] - 2024-1-18

Added

• Added setup_model_monitoring parameter to AutoMLOps.generate and orchestration/configs.py
• Added the creation of a model_monitoring/ directory and create_model_monitoring_job.sh script to generate
• Added a new AutoMLOps.monitor function, along with relevant templates and tests
• Added in a monitoring section into the generated config/defaults.yaml file
• Added logging.googleapis.com to the list of potentially required apis.
• Created 2 new functions for giving warnings when running AutoMLOps.monitor: get_model_monitoring_min_permissions and get_model_monitoring_recommended_roles in utils.py

Changed

• Updated the services/submission_service/main.py.j2 to include elements for automatic retraining based on monitoring anomaly logs, and adding in labels to the submit.
• Updated services/submission_service/requirements.txt to include google-cloud-storage.
• Updated the README.md.j2 to reflect the optional creation of the new model_monitoring/ directory.
• Changed the format for how config/defaults.yaml file gets written; this file is now written using the write_yaml_file function (yaml.safe_dump) in utils.py.
• Updated the account_permissions_warning function in utils.py to include a new operation: operation='model_monitoring'
• Updated the introductory example, readme, and user guide to reflect the new monitoring capabilities.
• Renamed validate_schedule in utils.py to validate_use_ci to reflect new requirements for model monitoring.

Fixed

[Akhan221]

Overall PR looks good to me! One thing to note, I tried running an example notebook locally and during the generate step it throws an error stating that _generate() got an unexpected keyword argument 'setup_modelmonitoring' despite uninstalling and reinstalling AMO. Let's have someone else pull down the branch and see if it works on their machine.

[Akhan221]

Resolved prior issue by using pip install .. However, now I get an import error: ImportError from google.cloud import logging. Need to add google-cloud-logging package to requirements.txt.

Also got this error: TypeError: Parameter to MergeFrom() must be instance of same class: expected google.cloud.aiplatform.v1.ModelDeploymentMonitoringScheduleConfig got ScheduleConfig. .

This seems to be a package conflict issue, need to make sure proper versions are set in requirements.txt.

[srastatter]

AutoMLOps.generate(...setup_model_monitoring=true) will create a directory AutoMLOps/model_monitoring that inside of it has the requirements.txt that you need to install in order to use this feature. I don't want to make these requirements come bundled with AutoMLOps by default.

Regarding the other issue, I think it's a package mismatch issue on your local machine - what version of google-cloud-aiplatform are you running?