The tmp folder holding the json and/or parquet files is always left on disk after upload_to_gcs is concluded. Sometimes it remains full of files - filling up the memory, which will cause an error eventually.
clean_up_task (or remove_tree in a previous attempt) is supposed to take care of it, but is implemented poorly.
The
tmpfolder holding the json and/or parquet files is always left on disk afterupload_to_gcsis concluded. Sometimes it remains full of files - filling up the memory, which will cause an error eventually.clean_up_task(orremove_treein a previous attempt) is supposed to take care of it, but is implemented poorly.