Currently the path naming for product files in the Storage Manager's backend storage is very simple: <backend bucket>/<product database ID> (e. g. s3://mybucket/456789 for S3 or /proseo/backend/456789 for POSIX). In a large system this may result in millions of products residing in the same directory, potentially exceeding the available system limits.
The path shall therefore be created in a manner which guarantees better partitioning of the backend storage. Since no prosEO service makes any assumptions as to how the product file path is generated (apart from an eventual s3:// protocol tag), the Storage Manager is free to apply any path naming convention it sees appropriate.
It is therefore proposed to change the path name to <backend bucket>/<YYYY>/<MM>/<DD>/<HH>/<product database ID>, with YYYY/MM/DD being the current date of the upload and HH the current hour. This will result in directories containing not more than perhaps a few hundred product directories in a Systematic Production scenario.
Currently the path naming for product files in the Storage Manager's backend storage is very simple:
<backend bucket>/<product database ID>(e. g.s3://mybucket/456789for S3 or/proseo/backend/456789for POSIX). In a large system this may result in millions of products residing in the same directory, potentially exceeding the available system limits.The path shall therefore be created in a manner which guarantees better partitioning of the backend storage. Since no prosEO service makes any assumptions as to how the product file path is generated (apart from an eventual
s3://protocol tag), the Storage Manager is free to apply any path naming convention it sees appropriate.It is therefore proposed to change the path name to
<backend bucket>/<YYYY>/<MM>/<DD>/<HH>/<product database ID>, withYYYY/MM/DDbeing the current date of the upload andHHthe current hour. This will result in directories containing not more than perhaps a few hundred product directories in a Systematic Production scenario.