Implement a simplified version of the current transform pipeline for Custom Vision Model. The current model pipeline performs mosaic, crop, convert and tiling as a series of spark jobs. This translates to separate spark-submit to the spark cluster in Synapse. Every spark-submit exacts an overhead of 2 to 4 minutes. This adds significant lead time for each of the transform - mosaic, crop, convert and tiling.
As a simple update, implement a consolidate version of the transform where the four transform are submitted as a single spark job. This reduces the redundant lead time associated with having multiple spark jobs.
Implement a simplified version of the current transform pipeline for Custom Vision Model. The current model pipeline performs mosaic, crop, convert and tiling as a series of spark jobs. This translates to separate spark-submit to the spark cluster in Synapse. Every spark-submit exacts an overhead of 2 to 4 minutes. This adds significant lead time for each of the transform - mosaic, crop, convert and tiling.
As a simple update, implement a consolidate version of the transform where the four transform are submitted as a single spark job. This reduces the redundant lead time associated with having multiple spark jobs.