Fix log file deletion/write mode

ccao-data / service-spark-iasworld

Service for extracting tables from the CCAO system-of-record and uploading them to the Data Department's data warehouse

GNU Affero General Public License v3.0

0 stars 0 forks source link

Fix log file deletion/write mode #14

Closed dfsnow closed 1 month ago

dfsnow commented 1 month ago

We had some Spark alarms this weekend indicating that our jobs didn't run. They actually did, but the logs weren't written to CloudWatch due to changes from 2212877054a983ad484d735b01aeae1163cc8cee. This PR reverts those changes and moves the log removal to the beginning of the job removal script.