Is your feature request related to a problem? Please describe.
Brickflow does not support the spark_jar_task feature yet . this is needed for any databricks jobs which gets executed using jar file
Cloud Information
[ X ] AWS
[ ] Azure
[ ] GCP
[ ] Other
Describe the solution you'd like
Need to enable a task called spark_jar_task in task.py
example
@wf.spark_jar_task(libraries=[JarTaskLibrary(jar="dbfs:<location>.jar or s3://<location>.jar")])
def example_jar():
return SparkJarTask(
main_class_name="com.example.Main",
)
Describe alternatives you've considered
tried using a bash operator but the solution is too complex
Is your feature request related to a problem? Please describe. Brickflow does not support the spark_jar_task feature yet . this is needed for any databricks jobs which gets executed using jar file
Cloud Information
Describe the solution you'd like Need to enable a task called spark_jar_task in task.py example
Describe alternatives you've considered tried using a bash operator but the solution is too complex