Store file metadata information in a file catalog
To get the prerequisites necessary for the file catalog:
pip install -r requirements.txt
To start an instance of the server running:
python -m file_catalog
All configuration is done using environment variables. To get the list of possible configuration parameters and their defaults, run
python -m file_catalog --show-config-spec
The primary interface is an HTTP server. TLS and other security hardening mechanisms are handled by a reverse proxy server as for normal web applications.
Requests to the main url /
are browsable like a standard website.
They will use javascript to activate the REST API as necessary.
Requests with urls of the form /api/RESOURCE
can access the
REST API. Responses are in HAL
JSON format.
uuid
(provided by File Catalog)logical_name
locations
(with at least one non-empty URL)file_size
checksum.sha512
/api/files
Resource representing the collection of all files in the catalog.
GET
Obtain list of files
limit
start
logical_name
(shortcut parameter)directory
(shortcut parameter)filename
(shortcut parameter)logical-name-regex
(shortcut parameter)run_number
(shortcut parameter)dataset
(shortcut parameter)event_id
(shortcut parameter)processing_level
(shortcut parameter)season
(shortcut parameter)query
keys
all-keys
(shortcut parameter)max_time_ms
200
: Response contains collection of file resources400
: Bad request (query parameters invalid)429
: Too many requests (if server is being hammered)500
: Unspecified server error503
: Service unavailable (maintenance, etc.)POST
Create a new file or add a replica
If a file exists and the checksum is the same, a replica is added. If the checksum is different a conflict error is returned.
200
: Replica has been added. Response contains link to file resource201
: Response contains link to newly created file resource400
: Bad request (metadata failed validation)409
: Conflict (if the file-version already exists); includes link to existing file429
: Too many requests (if server is being hammered)500
: Unspecified server error503
: Service unavailable (maintenance, etc.)DELETE
Not supported
PUT
Not supported
PATCH
Not supported
/api/files/{uuid}
Resource representing the metadata for a file in the file catalog.
GET
Obtain file metadata information
200
: Response contains metadata of file resource404
: Not Found (file resource does not exist)429
: Too many requests (if server is being hammered)500
: Unspecified server error503
: Service unavailable (maintenance, etc.)POST
Not supported
DELETE
Delete the metadata for the file
204
: No Content (file resource is successfully deleted)404
: Not Found (file resource does not exist)429
: Too many requests (if server is being hammered)500
: Unspecified server error503
: Service unavailable (maintenance, etc.)PUT
Fully update/replace file metadata information
200
: Response indicates metadata of file resource has been updated/replaced404
: Not Found (file resource does not exist) + link to “files” resource for POST409
: Conflict (if updating an outdated resource - use ETAG hash to compare)429
: Too many requests (if server is being hammered)500
: Unspecified server error503
: Service unavailable (maintenance, etc.)PATCH
Partially update/replace file metadata information
The JSON provided as body to PATCH need not contain all the keys, only the need to be updated. If a key is provided with a value null, then that key can be removed from the metadata.
200
: Response indicates metadata of file resource has been updated/replaced404
: Not Found (file resource does not exist) + link to “files” resource for POST409
: Conflict (if updating an outdated resource - use ETAG hash to compare)429
: Too many requests (if server is being hammered)500
: Unspecified server error503
: Service unavailable (maintenance, etc.)limit
limit
parameter. In cases where the server does not honor the limit
parameter, it should do so by providing fewer resources (limit
should be considered the client’s upper limit for the number of resources in the response).start
start
parameterstart
by limit
to paginate through many resultsquery
keys
|
-delimited string-list of keys; defines what fields to include in result(s)"foo|bar|baz"
all-keys
max_time_ms
None
; timeout to kill long queries in MILLISECONDSNone
indicates no timeout (this can hang the server -- you have been warned)logical-name-regex
, logical_name
, directory
, filename
In decreasing order of precedence...
logical-name-regex
query: {"logical_name": {"$regex": p}}
logical_name
query["logical_name"]
directory
query: {"logical_name": {"$regex": "^/your/path/.*"}}
/
will be inserted if you don't provide onefilename
(ie: /root/dirs/.../filename
)filename
query: {"logical_name": {"$regex": ".*/your-file$"}}
/
will be inserted if you don't provide onedirectory
(ie: /root/dirs/.../filename
)run_number
query["run.run_number"]
dataset
query["iceprod.dataset"]
event_id
query: {"run.first_event":{"$lte": e}, "run.last_event":{"$gte": e}}
processing_level
query["processing_level"]
season
query["offline_processing_metadata.season"]
all-keys
True
/"True"
/"true"
/1
); include all fields in result(s)/api/files
, grabbing the uuid, then querying /api/files/{uuid}
Follow these steps to get a development environment for the File Catalog:
cd ~/projects
git clone git@github.com:WIPACrepo/file_catalog.git
cd file_catalog
./setupenv.sh
This command will spin up a disposable MongoDB instance using Docker:
docker run \
--detach \
--name test-mongo \
--network=host \
--rm \
circleci/mongo:latest-ram
The following commands will create a Docker container for the file-catalog:
docker build -t file-catalog:{version} -f Dockerfile .
docker image tag file-catalog:{version} file-catalog:latest
Where {version} is found in file_catalog/initpy; e.g.:
__version__ = '1.2.0' # For {version} use: 1.2.0
Here are some commands to get the Docker container pushed to our Docker register in our Kubernetes cluster:
kubectl -n kube-system port-forward $(kubectl get pods --namespace kube-system -l "app=docker-registry,release=docker-registry" -o jsonpath="{.items[0].metadata.name}") 5000:5000 &
docker tag file-catalog:{version} localhost:5000/file-catalog:{version}
docker push localhost:5000/file-catalog:{version}