added 04g and updated script to use aip predef variable for model saves

[statmike]

Hey @goodrules , This series is becoming amazing! Thank you for all the effort going into this. I have a suggestion that you can request we push to a future pull but wanted to mention here in case you want to go ahead and incorporate it:

The two script files: train.py and hp_train.py

• now that you are using the environment variable AIP_MODEL_DIR you can do a few extra things...
• it looks like the following args can be removed now: bucket, uri, blob, timestamp. This will impact the scripts and the notebooks that use the scripts will need an update for CMDARGS.
• training jobs automatically mount gcs at the path /gcs. You can treat gcs locations like folders (a few extra details about directly writing... might need to create files then copy in).
  • you could remove the need for the gcs storage client
  • write the files then copy to gcs using the fuse mount folder by
    • some file types might be able to be written directly...
    • use a method like shutil.copyfile(src, dest) to do a copy job
  • parse AIP_MODEL_DIR and remove the prefix gs://. Then the path can be referenced as a folder: gcs/bucket/path/to/file.ext. Something like f'gcs/{AIP_MODEL_DIR[5:]}/filename.ext' would work for the directory location.
  • reference link for fuse in Vertex AI

Let me know if this makes sense. I think this will both clean the workflow up a little and showcase the value of the GCS Fuse mount that is automatic with training jobs! Thank You, @statmike

[goodrules]

makes sense. will make the changes this week

[goodrules]

@statmike should be good for merge. added 04i and updated the rest with the suggestions above