Fix bug in changing permissions on downloaded files

[xylar]

The code for changing permissions was being given an incorrect directory and then was failing silently. This merge fixes the path and also removes the silent failure (an error will be raised if the status of the download directory cannot be determined).

This merge also adds the database root and the root for each affected core to the list of directories to chmod/chown anytime any files get downloaded.

A port of https://github.com/MPAS-Dev/compass/pull/609

Checklist

• [x] Testing comment in the PR documents testing used to verify the changes

[xylar]

Testing

I set up the cosine_bell_cached_init suite on Perlumtter and verified that the resulting files and directories all had the expected permissions:

/global/cfs/cdirs/e3sm/polaris> ls -lah */*/*/*/*/*/*/
ocean/polaris_cache/global_convergence/icos/cosine_bell/Icos120/init/:
total 57M
drwxrwxr-x 2 xylar e3sm 4.0K Apr 18 17:08 .
drwxrwxr-x 4 xylar e3sm 4.0K Apr 18 17:08 ..
-rw-rw-r-- 1 xylar e3sm  57M Apr 18 17:08 initial_state.230220.nc

ocean/polaris_cache/global_convergence/icos/cosine_bell/Icos120/mesh/:
total 49M
drwxrwxr-x 2 xylar e3sm 4.0K Apr 18 17:08 .
drwxrwxr-x 4 xylar e3sm 4.0K Apr 18 17:08 ..
-rw-rw-r-- 1 xylar e3sm 523K Apr 18 17:08 cellWidthVsLatLon.230220.nc
-rw-rw-r-- 1 xylar e3sm 1.4M Apr 18 17:08 graph.230220.info
-rw-rw-r-- 1 xylar e3sm 4.0M Apr 18 17:08 mesh.230220.msh
-rw-rw-r-- 1 xylar e3sm  43M Apr 18 17:08 mesh.230220.nc
...