Add the minimum CLI version supported by the template

[shreyas-goenka]

This CLI adds minimum version of the CLI required to use the template. Newer features (eg: https://github.com/databricks/mlops-stacks/pull/112) used by the stack are not compantible with older CLI versions.

The CLI respects the min_databricks_cli_version metadata field and returns an error to the user.

Here's the output for a couple different CLI versions: CLI version >= {min_databricks_cli_version} (in this case v0.209): Works as expected

shreyas.goenka@THW32HFW6T playground % ./databricks_cli_0.209.0_darwin_arm64/databricks bundle init ~/mlops-stack
Welcome to MLOps Stacks. For detailed information on project generation, see the README at https://github.com/databricks/mlops-stacks/blob/main/README.md. 

Project Name [my-mlops-project]: ^C

CLI version < {min_databricks_cli_version} AND >= v0.208.2 (when support for min_databricks_cli_version was introduced)

shreyas.goenka@THW32HFW6T playground % ./databricks_cli_0.208.2_darwin_arm64/databricks bundle init ~/mlops-stack
Error: minimum CLI version "v0.209.0" is greater than current CLI version "v0.208.2". Please upgrade your current Databricks CLI

CLI version == v0.208.2 Unfortunately we don't have any method to display an error to the users of this version of the CLI.

CLI version <= v0.208.1 Hacky solution to give warnings to users of older CLI versions. This is the best effort we can do. Once the CLI goes GA this hack can be reverted.

shreyas.goenka@THW32HFW6T playground % ./databricks_cli_0.208.0_darwin_arm64/databricks bundle init ~/mlops-stack
{{if false}}

ERROR: This template is no longer supported supported by CLI versions v0.209 and lower.
Please hit control-C and go to https://docs.databricks.com/en/dev-tools/cli/install.html for instructions on upgrading the CLI.

{{end}}Welcome to MLOps Stacks. For detailed information on project generation, see the README at https://github.com/databricks/mlops-stacks/blob/main/README.md. 

Project Name [my-mlops-project]: 

[mingyu89]

It works but it's a very hacky way. Do we have better options?

[arpitjasa-db]

@shreyas-goenka if we don't have this hacky solution for CLI version < v0.208.1, what error do we run into?

[shreyas-goenka]

Customers using a CLI with version < 0.209 will run into issues once we use enums and start using the selection UI for those prompts. Older versions of the CLI did not deal with multiline prompts correctly for selection type questions and would end up overlaying on top of already printed text.

Here's an example of what the experience would look like. https://github.com/databricks/mlops-stacks/assets/88374338/847668d4-3982-49d3-aeb3-682cf386310a

I agree that this method is hacky and does not look professional. However, I could not come up with any other alternatives and this is probably the best effort we can do here and the right thing for our customers. We made an oversight by not including versioning right out of the box for templates. I'll defer to your judgement however on whether we should include the hacky deprecation notice.

[arpitjasa-db]

Customers using a CLI with version < 0.209 will run into issues once we use enums and start using the selection UI for those prompts. Older versions of the CLI did not deal with multiline prompts correctly for selection type questions and would end up overlaying on top of already printed text.

Here's an example of what the experience would look like. https://github.com/databricks/mlops-stacks/assets/88374338/847668d4-3982-49d3-aeb3-682cf386310a

I agree that this method is hacky and does not look professional. However, I could not come up with any other alternatives and this is probably the best effort we can do here and the right thing for our customers. We made an oversight by not including versioning right out of the box for templates. I'll defer to your judgement however on whether we should include the hacky deprecation notice.

Love the music choice for that video 😂 but yeah I see the issue, it doesn't fail gracefully with older versions. Is there any way we can send an interrupt ourselves?

[mingyu89]

Cookiecutter has the support of private variable that does not take input from users and has the option to show/not show the private variable values in console output. https://cookiecutter.readthedocs.io/en/stable/advanced/private_variables.html

How about adding similar support in bundles? Either by using a convention in the name, or adding an explicit field. After adding the support, we can add the private variable as the first one for prompt. At least it's in a separate input variable.

[shreyas-goenka]

@mingyu89 That's a good suggestion and gets rid of the need for a hack. The main hesitation I have with this approach is we should not introduce a new feature to bundle templates (in this case private variables) for it's side effects related to having a smooth migration path here.

[mingyu89]

Yeah if it's not a feature we wanted to add, nevermind about it.