save_axs_table() deletes files from path

astronomy-commons / axs

Astronomy eXtensions for Spark: Fast, Scalable, Analytics of Billion+ row catalogs

https://axs.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

23 stars 12 forks source link

save_axs_table() deletes files from path #6

Open ctslater opened 5 years ago

ctslater commented 5 years ago

I was using save_axs_table() to save the new ZTF data and used the path= argument to put it on the new disk drives. Spark then deleted everything from the path I gave it, including a bunch of unrelated directories and all the ZTF data I was trying to import.

zecevicp commented 5 years ago

This is how Spark's saveAsTable() method works. I added a warning note in the save_axs_table() method's documentation about this.

ctslater commented 5 years ago

Let's leave this open until we have more of a safeguard. I agree that it's how spark works, but I lost a few days of work to this and it could be much worse if someone specifies their home directory, for example, so I think it's a very risky thing to expose to the user.