aspark21
commented
2 years ago Quick update: We had thought about this previously and now I recall what the concern was: The setting enabletasks needs to be enabled for any of the tasks to run. But by default that runs all of the tasks - the upload and pull ones included. And that doesn't make sense if we don't want to start uploading files to S3 - since they are already local & external or local-only.
Which is why we hadn't set enabletasks on the "a snapshot copy of prod once all the files are already in S3 " use case.
Likewise in this use case as we only want to do check_objects_location initially (and generate_status_report to get an overview report of status) and only once fully checked, start running delete_local_objects.
After some actual digging this time, what we'd do is: 1) Disable all of the objectfs scheduled tasks, in particular: \tool_objectfs\task\push_objects_to_storage \tool_objectfs\task\pull_objects_from_storage \tool_objectfs\task\delete_local_objects \tool_objectfs\task\delete_local_empty_directories \tool_objectfs\task\recover_error_objects
2) Enable Tasks to run tool_objectfs | enabletasks => Yes
3) Run the \tool_objectfs\task\check_objects_location Task
Either by having task running when site cron active or by running that task on it's own cron via
php admin/cli/scheduled_task.php --execute='\tool_objectfs\task\check_objects_location'
And a daily cron one for
php admin/cli/scheduled_task.php --execute='\tool_objectfs\task\generate_status_report'
4) Start local deletions once checks complete
Either by having task running when site cron active or by running that task on it's own cron via
php admin/cli/scheduled_task.php --execute='\tool_objectfs\task\delete_local_objects'
5) Once shared data cleared from local filedir Leave as is essentially as the deletion task will ensure we only keep requested files temporarily and disk usage doesn't grow back
So I think our core use case for "existing historical snapshots which are 100% on local disk but for which the files are already in S3 for our prod (i.e. snapshot taken before prod went cloud/S3)" is already covered, it just needs configuring appropriately.
If it would be of use, I could try and write that up into the documentation for others to refer to in future.
The one bit that isn't covered is this: -optionally, upload files which don't exist in the prod S3 into a separate S3 bucket (so we can delete those once we do away with those older snapshots)
Which we can probably come back to once we've reduced disk usage with the above.
That kind of made me wonder about a future where files are in one storage and there is a desire to move them into a different storage engine, for whichever reason moving from one cloud to another. The mechanics are there but both of those features would be quite big things to implement.
brendanheywood
The Use Cases defined in the readme including use of old files and re-use in multiple environments. This currently works well for prod, a snapshot copy of prod once all the files are already in S3 and all sorts of testing instances which can be given read-only access to S3 to have a fully working instance.
We are looking at applying this to a very similar use case, but which does bring some quirks: existing historical snapshots which are 100% on local disk but for which the files are already in S3 for our prod (i.e. snapshot taken before prod went cloud/S3).
That means it doesn't need to upload files, just go looking to see wether they already exist.
What we'd essentially like to do is: -give the snapshots read-only credentials to the prod S3 bucket -enable objectfs to find the local files which exist in S3 -delete the local copies -don't upload files which are only local to the prod S3 -optionally, upload files which don't exist in the prod S3 into a separate S3 bucket (so we can delete those once we do away with those older snapshots)
We have 6 snapshots, with the same files existing multiple times. In total, currently using 25TB of storage, this would be a massive space/cost saver.
While it feels from the naming of the various tasks that this is not quite how objectfs would behave, it wight actually already be there. Can pick up in WRMS what would need developing/funding to get there.