Closed mcast closed 4 weeks ago
On the command line, the way I would achieve what you're describing is to preprocess the JSON document being sent to baton-metasuper
to remove the properties that you don't want to modify, e.g. by passing it through jq
.
For most of our complex metadata editing we use an API built on the icommands and baton
. If you're using Perl, then perhaps it might be worth considering. It is rather connection-hungry, so may not be suitable if that's a concern.
I have also encountered the same issue as @mcast.
To do a set (as described by @mcast), it is necessary to do 3 operations:
@colin-nolan not forgetting step 4. hope very much that no other proess was doing the same thing with different changes!
We're using iRODS to implement a big bucket of files and decided to take the "patch over uncertainty and inconvenience by adding another layer" approach. So far I avoided the problem by
baton-metaset
only on freshly created files, i.e. empty set of existing keys._set_AVUs_exclusively
in my API and it's called during put_file
.imeta set -d
rather sparingly where I need it._set_AVUs
so could convert it to Baton later if that was appropriate.My only use of _set_AVUs
so far is to add one AVU, so probably no gain unless I need to call it repeatedly on multiple files (quite possible) and I fed Baton a stream of jsonl/ndjson.
Adding a set
operation is a very worthwhile enhancement and I'll add it to the next milestone. That will have the same semantics as imeta set
, which fits what you're looking for.
baton
is by design a thin layer over iRODS which allows you to talk JSON as a stream, while preserving standard iRODS behaviours. Multi-valued attributes and non-idempotence of the add
operation are intrinsic to iRODS and therefore I will not change these.
With respect to
'4. hope very much that no other process was doing the same thing
That sentiment applies to anything done with iRODS metadata as currently implemented (certainly up to 4.1.9) because the ICAT behaves as if both AUTOCOMMIT is enabled and no transactions used. There is no way to prevent other clients getting caught in race conditions with your operations.
Closed as inactive.
I was looking for something between the two existing baton-meta* commands, more like
imeta set -d
...It's just a suggestion which I hope is useful for the future. I'm expecting to go ahead with
imeta set -d
and take the multiple connect overheads in parallel across files.Problems with
baton-metamod --operation add
Problem with
baton-metasuper
Continuing from above,
The smippy AVU has been lost.
Suggested enhancement
Continuing from above,
Actions
flumm: P
and no-op forele: phant
.spram: fubil
(and any otherspram
keys, if there were more than one) and replaceele: phant
withele: baroo
.I'm assuming that iRODS itself can't store a null value, so we're not losing anything by extending the meaning of the
null
value to mean deletion.Rationale
The reason I would like to supersede only specified values is that I have namespaced my keys.
I would like to be able to set (add, no-op or replace) the keys with names beginning
test-mca-foo.
without touching any keys with names beginningtest-mca-bar.
The lazier and currently more likely reason is that I want to set
test-mca-foo.id_ifile
andtest-mca-foo.id_analysis_proc
in one bite, and then settest-mca-foo.ftrk_microstat
independently without having to re-calculate the intended values of the other keys in thetest-mca-foo.
group.