PelicanPlatform / xrootd-s3-http

An XRootD plugin that allows Pelican to interface with s3/http server backends
Apache License 2.0
2 stars 6 forks source link

HTTP backend does not support multiple exports #48

Open jhiemstrawisc opened 1 month ago

jhiemstrawisc commented 1 month ago

In the S3 plugin, we developed the special s3.begin and s3.end config directives so that we could point various namespaces at different buckets, but I don't think we have a similar mechanism for the HTTP plugin.

Consider the scenario, where the HTTP server at https://data.lhncbc.nlm.nih.gov/ hosts two image datasets, one with path /public/Tuberculosis-Chest-X-ray-Datasets/Montgomery-County-CXR-Set/MontgomerySet/CXR_png and one with path /public/Tuberculosis-Chest-X-ray-Datasets/Shenzhen-Hospital-CXR-Set/CXR_png. It would be nice to add the config blocks to http to accomplish a config similar to:

httpserver.begin
httpserver.url_base https://data.lhncbc.nlm.nih.gov/public/Tuberculosis-Chest-X-ray-Datasets/Montgomery-County-CXR-Set/MontgomerySet/CXR_png
httpserver.storage_prefix /prefix-1
httpserver.token_file <some token file to use with requests to that path, if provided>
httpserver.end

httpserver.begin
httpserver.url_base https://data.lhncbc.nlm.nih.gov/public/Tuberculosis-Chest-X-ray-Datasets/Shenzhen-Hospital-CXR-Set/CXR_png
httpserver.storage_prefix prefix-2
httpserver.token_file <some token file to use with requests to that path, if provided
httpserver.token_file <some token file to use with requests to that path, if provided>
httpserver.end

This would let us scope these two paths under separate prefixes, so a request for /prefix-1/foo.png turns into an HTTP request for https://data.lhncbc.nlm.nih.gov/public/Tuberculosis-Chest-X-ray-Datasets/Montgomery-County-CXR-Set/MontgomerySet/CXR_png/foo.png, while a request for /prefix-2/foo.png turns into an HTTP request for https://data.lhncbc.nlm.nih.gov/public/Tuberculosis-Chest-X-ray-Datasets/Shenzhen-Hospital-CXR-Set/CXR_png/foo.png

Note that I've also moved the httpserver.token_file directive into the begin/end blocks with the assumption that different http paths could require different access tokens.