search

mozilla / leanplum-data-export

A docker image that runs a script to export Leanplum data into BigQuery
5 stars 4 forks source link

Delete existing data before running #9

Closed fbertsch closed 4 years ago

fbertsch commented 4 years ago

If the job fails before deleting the GCS data, it will remain for the next run. We should delete the data before running to ensure that data isn't loaded multiple times.

BenWu commented 4 years ago

Isn't that what this line does? https://github.com/mozilla/leanplum-data-export/blob/862ef4adbc993d7af9965fd43cb654a9f2e87c47/leanplum_data_export/export.py#L94

Doesn't the save_files function delete the relevant files in the bucket before uploading the new ones?

fbertsch commented 4 years ago

Isn't that what this line does?

https://github.com/mozilla/leanplum-data-export/blob/862ef4adbc993d7af9965fd43cb654a9f2e87c47/leanplum_data_export/export.py#L94

Doesn't the save_files function delete the relevant files in the bucket before uploading the new ones?

It definitely is. Looks like I was a bit late on this one :)