Feature run job task, sparkJAR and SQL task

[RajuGujjalapati]

Description

Added code to support Run Job, JAR and SQL task types. Run Job task can trigger another job by its job_name, JAR task will trigger by providing jar file as library and its main class and SQL tasks will be useful to execute SQL objects such as queries, legacy dashboards, files and alerts

Related Issue

Link to the issue: JAR #49 Link to the issue: RunJob #55 Link to the issue: SQL #119

Motivation and Context

1. Databricks now supports creating a "Run Job" task, which can trigger another job by its job_id, its good to have this feature in brickflow.
2. Databricks now supports creating a "JAR" task. Using the JAR task, you can ensure fast and reliable installation of Java or Scala code in your Databricks jobs.
3. Databricks now supports creating a "SQL" task. Using the SQL task, SQL tasks will be useful to execute SQL objects such as queries, legacy dashboards, files and alerts

How Has This Been Tested?

Unit tests + deployed test workflow using wheel file

Screenshots (if appropriate):

1. All test cases passed:

   

2. Tested using brickflow-bundles-test:

4. Test workflow with a JAR task created using whl file:
5. Test workflow with a Run Job task created using whl file:
6. Test workflow with a SQL job task created using whl file:

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [x] I have updated the documentation accordingly.
• [x] I have read the CONTRIBUTING document.
• [x] I have added tests to cover my changes.
• [x] All new and existing tests passed.