delete batch job: unable to remove entire job directory

[bossie]

Integration test test_random_forest_train_and_load_from_jobid:

• training job T creates randomforest.model.tar.gz;
• inference job I loads T's model (unzips to randomforest.model in T's job directory);
• deleting T logs a warning as it is unable to to delete randomforest.model (because only the owner of the directory has write permissions to do so and the web app user doesn't), r-2407093f291a46da8ea7cf50793a99da:

Could not recursively delete /data/projects/OpenEO/j-240709c75fd04a1c9241f24c8264ab6e

Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/backend.py", line 2859, in _delete_job
    shutil.rmtree(job_dir)
  File "/usr/lib64/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib64/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib64/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib64/python3.8/shutil.py", line 675, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib64/python3.8/shutil.py", line 673, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: '._SUCCESS.crc'

[bossie]

Same for a batch job that runs a UDF with a dependency in it:

{
  "process_graph": {
    "loadcollection1": {
      "process_id": "load_collection",
      "arguments": {
        "bands": [
          "B02",
          "B03"
        ],
        "id": "SENTINEL2_L2A",
        "spatial_extent": {
          "west": 4,
          "south": 51,
          "east": 4.01,
          "north": 51.01
        },
        "temporal_extent": [
          "2023-09-01",
          "2023-09-15"
        ]
      }
    },
    "apply1": {
      "process_id": "apply",
      "arguments": {
        "data": {
          "from_node": "loadcollection1"
        },
        "process": {
          "process_graph": {
            "runudf1": {
              "process_id": "run_udf",
              "arguments": {
                "data": {
                  "from_parameter": "x"
                },
                "runtime": "Python",
                "udf": "\n# /// script\n# dependencies = [\"duviz\"]\n# ///\n\nimport re\nimport xarray\nimport duviz\n\ndef apply_datacube(cube: xarray.DataArray, context: dict) -> xarray.DataArray:\n    # Get a step size by counting the number of things \n    # in the duviz module having a name with \"u\" in it.\n    step = sum(\"u\" in x for x in dir(duviz))\n    # Zero out pixels every `step` along x axis\n    cube[{\"x\": slice(None, None, step)}] = 0 \n    return cube\n"
              },
              "result": true
            }
          }
        }
      }
    },
    "saveresult1": {
      "process_id": "save_result",
      "arguments": {
        "data": {
          "from_node": "apply1"
        },
        "format": "netCDF",
        "options": {}
      },
      "result": true
    }
  }
}

Deleting this job, r-24070953694445ff86db16e6757dba9b:

Could not recursively delete /data/projects/OpenEO/j-2407097ceb824b6397b203abade3f6e3

Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/backend.py", line 2859, in _delete_job
    shutil.rmtree(job_dir)
  File "/usr/lib64/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib64/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib64/python3.8/shutil.py", line 675, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib64/python3.8/shutil.py", line 673, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'duviz.py'