Invalid files in data_object_set

shreddd commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

I'm trying to re-link the Globus files to capture the current data in NMDC, and we found a bunch of files in Mongo that don't seem to exist on the filesystem.

If you look on CFS in: /global/homes/n/nmdcda/infra-admin/globus/check_files.log There are a number of missing files and dirs (look for ERROR or WARN).

We should try and figure out if

these are just obsoleted data objects in Mongo (and whether these should be deleted)
if there are missing or mislabeled files or if we have incorrect references

This is what we get from running https://github.com/microbiomedata/infra-admin/blob/main/globus/inspect_live_directories.py which queries all records in data_object_set and extracts URLs from them.

Expected Behavior

No response

Steps To Reproduce

This is what we get from running https://github.com/microbiomedata/infra-admin/blob/main/globus/inspect_live_directories.py which queries all records in data_object_set and extracts URLs from them.

Output in: /global/homes/n/nmdcda/infra-admin/globus/check_files.log

Environment

No response

Anything else?

No response

aclum commented 1 month ago

@shreddd How long does it take to run inspect_live_directories.py?

aclum commented 1 day ago

How exactly was this run? When i run with python inspect_live_directories.py check_files_and_directories --base-api-url https://data.microbiomedata.org/ --base-data-directory /global/cfs/cdirs/m3408/results I get requests.exceptions.JSONDecodeError

microbiomedata / issues