hive catalog drop table XX purge not delete hdfs path

[mengshangxun]

Apache Iceberg version

1.4.3 (latest release)

Query engine

Spark

Please describe the bug 🐞

hdfs+spark3.3+iceberg1.4.3+hive catalog 1、create table test(id string,name string) using iceberg; 2、insert into test select 'a','a'; 3、drop table test purge; purge sql only delete iceberg datafile and metadata file，but hdfs path still exist. it is confused when reuse this path.

[manuzhang]

This is expected behavior as Iceberg only purges its referenced metadata and data, and nothing more.

[nqvuong1998]

In my case, when using PURGE, Iceberg only deletes metadata files. Do I need any conf to be able to delete data files?

[RussellSpitzer]

No it also removes data files, the only exception is files which are not referenced by the Iceberg table.

[mengshangxun]

when create a hive catalog table ,it will create hdfs path in the spark warehouse,but now neither drop table ordrop table purge can delete this path. if i use drop table and create table in the same path,this path will contain old meta file and data file. this old files will nerver delete. is delete path after delete files is a better choice?

[wanghualei]

it is a big problem!

[vinhnemo]

so which is the best way to clean the data when tables are dropped?