[BugFix] The hdfs directory is not synchronized when the spark resource is deleted

[blanklin030]

Steps to reproduce the behavior (Required)

• 1. create spark load

     LOAD LABEL pre_stream.test_load_ly_2 (
     DATA FROM TABLE test_list_dup_sr_external_h2s_foit_820240510
     INTO TABLE test_list_dup_sr
     TEMPORARY PARTITION(temp__p20230930_BR)
     SET (
     `id` = `id`,
     `name` = `name`,
     `dt` = '2023-09-30',
     `country_code` = 'BR'
     ) 
     )WITH RESOURCE 'spark_resource' (
     "spark.yarn.tags" = "xxx05131",
     "spark.dynamicAllocation.enabled" = "true",
     "spark.executor.memory" = "3g",
     "spark.executor.memoryOverhead" = "2g",
     "spark.streaming.batchDuration" = "5",
     "spark.executor.cores" = "1",
     "spark.yarn.executor.memoryOverhead" = "2g",
     "spark.speculation" = "false",
     "spark.dynamicAllocation.minExecutors" = "2",
     "spark.dynamicAllocation.maxExecutors" = "100"
     ) PROPERTIES (
     "timeout" = "72000",
     "spark_load_submit_timeout" = "7200"
     )
     ;

• 2. some directory is created

     
     2024-05-14 01:42:12,013 INFO (pending_load_task_scheduler_pool-1|498) [SparkRepository.upload():302] finished to upload file, localPath=/home/hadoop/starrocks-current/fe/spark-dpp/spark-dpp-1.0.0-jar-with-dependencies.jar, remotePath=hdfs://ClusterNmg/user/prod_xxx/sparketl/1384206915/__spark_repository__db__tb_sr__1019adb1d38c/__archive_1.0.0/__lib__spark-dpp-1.0.0-jar-with-dependencies.jar

2024-05-14 01:42:12,077 INFO (pending_load_task_scheduler_pool-1|498) [SparkRepository.rename():316] finished to rename file, originPath=hdfs://ClusterNmg/user/prod_xxx/sparketl/1384206915/spark_repositorydbtb_sr1019adb1d38c/archive_1.0.0/libspark-dpp-1.0.0-jar-with-dependencies.jar, destPath=hdfs://ClusterNmg/user/prod_xxx/sparketl/1384206915/spark_repositorydbtb_sr1019adb1d38c/archive_1.0.0/__lib_70688c469808112f344091125a860404_spark-dpp-1.0.0-jar-with-dependencies.jar

+ 3. drop spark resource

drop resource spark_resource

+ 4. The hdfs directory is not synchronized when the spark resource is deleted

[hadoop@bigdata-starrocks-xxx ~]$ hdfs dfs -ls hdfs://ClusterNmg/user/prod_xxx/sparketl/1384206915/spark_repository__spark_resource/archive_1.0.0/ Found 2 items -rw-r--r-- 3 prod_xxx supergroup 394653421 2024-05-20 10:54 hdfs://ClusterNmg/user/prod_xxx/sparketl/1384206915/spark_repository__spark_resource/archive_1.0.0/lib_62eff19a2751990e17b47aa258fb7623_spark-2x.zip -rw-r--r-- 3 prod_xxx supergroup 4013682 2024-05-20 10:53 hdfs://ClusterNmg/user/prod_xxx/sparketl/1384206915/spark_repositoryspark_resource/__archive_1.0.0/lib_70688c469808112f344091125a860404_spark-dpp-1.0.0-jar-with-dependencies.jar

### Expected behavior (Required)
drop spark resource and delete spark directory

### Real behavior (Required)
drop spark resource and the spark directory didn't remove

### StarRocks version (Required)
 - You can get the StarRocks version by executing SQL `select current_version()`

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!