lanl / dsi

LANL Data Science Infrastructure Project
https://lanl.github.io/dsi
5 stars 3 forks source link

Handle complex metadata permission storage #58

Open qwofford opened 1 year ago

qwofford commented 1 year ago

See #39

qwofford commented 10 months ago

The permissions metadata tracker should:

  1. By default, give an error on artifact_handler put requests that have column permissions metadata which union to a set of size > one.
  2. Given a param indicating "scary mode", save files out, one for each unique permissions configuration in the union of the columns permissions metadata. Preserve posix file security information in resulting files. Unit tests should validate permissions are are preserved for information involved. The user should have to explicitly select "y" to save data.
  3. Given a param indicating "very scary mode", save all data to the same file and do so with many loud warnings including a report of all the permissions data that were squashed during this operation, and what they were squashed to. The user should have to explicitly select "y" to save data.

The DSI documentation for back-end drivers should be extended to explain this behavior.

In a maximally conservative workflow, we would never save data, but always read data using plugins before incorporating query functionality.

Coordinate with @kchilleri on this, and feel free to create sub-issues as needed.

DanielRJohnson commented 8 months ago

Updated this branch to catch up with main and to implement the functionalities requested.

Some implementation notes:

No rush to review, holidays are coming up! :)

DanielRJohnson commented 8 months ago

@qwofford @kchilleri I can't add y'all as reviewers since Quincy is the owner or the PR and Krishna isn't listed, but see above ^

qwofford commented 8 months ago

Note that we should squash these commits because I made a mess of the git history while resolving conflicts.

qwofford commented 8 months ago

Looking excellent as usual. I'm adding some documentation for this new capability. When you @DanielRJohnson have an opportunity please take a look at what I wrote and we can discuss after the holiday:

https://lanl.github.io/dsi/permissions.html