we need to add stream back features for spark transform. When the treamBackFlag is true, we'll start a file source connector right away after sending the pyspark code to Livy. The regular spark job status updates daemon will check to see if the spark export data is done. If yes, it will start a file source connect and monitor, and delete it until all exported files are processed.
datafibers
commented
6 years ago
we need to add stream back features for spark transform. When the treamBackFlag is true, we'll start a file source connector right away after sending the pyspark code to Livy. The regular spark job status updates daemon will check to see if the spark export data is done. If yes, it will start a file source connect and monitor, and delete it until all exported files are processed.