kript
commented
1 year ago I should note that I get this behaviour when I add the metadata both with imeta and with iput --metadata
Closed kript closed 10 months ago
kript
commented
1 year ago I should note that I get this behaviour when I add the metadata both with imeta and with iput --metadata
alanking
commented
1 year ago We will look into it. My initial thought is that this is related to some changes which occurred in the trim API as a result of the logical locking effort, so that is where I would suggest we start looking.
trel
commented
1 year ago Just noting for future readers - "root tree has a replication passthru" is a confusing phrase... root only has a replication resource.
A passthru is different.
trel
commented
1 year ago I have configured 4.2.11 on an Ubuntu 18.04 container with the following settings...
Set up hierarchies...
irods@1b97fdcaad93:~$ ilsresc
demoResc:unixfilesystem
noReplRoot:passthru
└── blue:random
├── ufs3:unixfilesystem
└── ufs4:unixfilesystem
root:passthru
└── replicate:replication
├── ufs1:unixfilesystem
└── ufs2:unixfilesystemConfigured tiers...
irods@1b97fdcaad93:~$ imeta ls -R root
AVUs defined for resource root:
attribute: irods::storage_tiering::group
value: ega_transfer_group
units: 0
----
attribute: irods::storage_tiering::query
value: SELECT DATA_NAME, COLL_NAME, USER_NAME, DATA_REPL_NUM WHERE META_DATA_ATTR_NAME = 'tier:single-copy' AND META_DATA_ATTR_VALUE = '1' AND DATA_RESC_HIER like 'root;replicate%' AND DATA_ACCESS_TYPE >= '1120' AND USER_ZONE = 'tempZone' AND USER_TYPE = 'rodsadmin' and USER_NAME like 'irods%'
units:
----
attribute: irods::storage_tiering::time
value: 18000000000
units:
irods@1b97fdcaad93:~$ imeta ls -R noReplRoot
AVUs defined for resource noReplRoot:
attribute: irods::storage_tiering::group
value: ega_transfer_group
units: 1
----
attribute: irods::storage_tiering::maximum_delay_time_in_seconds
value: 30
units:
----
attribute: irods::storage_tiering::minimum_delay_time_in_seconds
value: 1
units:
----
attribute: irods::storage_tiering::minimum_restage_tier
value: true
units:
----
attribute: irods::storage_tiering::object_limit
value: 500
units:
----
attribute: irods::storage_tiering::verification
value: checksum
units: Installed the periodic rule...
irods@1b97fdcaad93:~$ cat sanger.r
{
"rule-engine-instance-name": "irods_rule_engine_plugin-unified_storage_tiering-instance",
"rule-engine-operation": "irods_policy_schedule_storage_tiering",
"delay-parameters": "<INST_NAME>irods_rule_engine_plugin-unified_storage_tiering-instance</INST_NAME><PLUSET>1s</PLUSET><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF>",
"storage-tier-groups": [
"ega_transfer_group"
]
}
INPUT null
OUTPUT ruleExecOut
irods@1b97fdcaad93:~$ irule -F sanger.r
irods@1b97fdcaad93:~$ iqstat
id name
10042 {"rule-engine-operation":"irods_policy_storage_tiering","storage-tier-groups":["ega_transfer_group"]} Then ... testing as a new irodstester rodsadmin account...
root@1b97fdcaad93:/# echo "hola" > hola
root@1b97fdcaad93:/# iput -R root hola
root@1b97fdcaad93:/# imeta add -d hola tier:single-copy 1
root@1b97fdcaad93:/# ils -L hola
irodstester 0 root;replicate;ufs1 5 2023-03-15.20:36 & hola
generic /tmp/ufs1vault/home/irodstester/hola
irodstester 1 root;replicate;ufs2 5 2023-03-15.20:36 & hola
generic /tmp/ufs2vault/home/irodstester/hola
root@1b97fdcaad93:/# ils -L hola; iqstat -a
irodstester 2 noReplRoot;blue;ufs3 5 2023-03-15.20:42 & hola
sha2:Ez7piSk/knNjASgMbxTInVISAMF9zc7MowzSBwUzLUQ= generic /tmp/ufs3vault/home/irodstester/hola
No delayed rules pendingIt worked! single replica. checksum. on the noReplRoot tier.
But the log was a bit noisy with a caught error due to the attempted migration of the second replica (duplicate key error when trying to move replica 2 since replica 3 was already created on the target resource)...
Not sure, but I think we've already cleaned this up since 4.2.11... can investigate separately...
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[1]=10040
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[2]=10038
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[3]=hola
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[4]=2
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[5]=
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[6]=generic
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[7]=5
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[8]=EMPTY_RESC_GROUP_NAME
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[9]=EMPTY_RESC_NAME
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[10]=EMPTY_RESC_HIER
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[11]=10023
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[12]=/tmp/ufs4vault/home/irodstester/hola
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[13]=irodstester
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[14]=tempZone
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[15]=2
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[16]=
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[17]=
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[18]=00000000000
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[19]=0
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[20]=0
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[21]=
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[22]=01678927367
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[23]=01678927367
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[24]=10023
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[25]=/tmp/ufs4vault/home/irodstester/hola
Mar 15 20:42:47 pid:7627 NOTICE: bindVar[26]=10040
Mar 15 20:42:47 pid:7627 NOTICE: _cllExecSqlNoResult: SQLExecDirect error: -1 sql:insert into R_DATA_MAIN ( data_id, coll_id, data_name, data_repl_num, data_version, data_type_name, data_size, resc_group_name, resc_name, resc_hier, resc_id, data_path, data_owner_name, data_owner_zone, data_is_dirty, data_status, data_checksum, data_expiry_ts, data_map_id, data_mode, r_comment, create_ts, modify_ts ) select ?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,? where not exists (select data_id from R_DATA_MAIN where resc_id=? and data_path=? and data_id=?)
Mar 15 20:42:47 pid:7627 NOTICE: SQLSTATE: S1010
Mar 15 20:42:47 pid:7627 NOTICE: SQLCODE: 0
Mar 15 20:42:47 pid:7627 NOTICE: SQL Error message: [unixODBC][Driver Manager]Function sequence error
Mar 15 20:42:47 pid:7627 NOTICE: SQLSTATE: 23505
Mar 15 20:42:47 pid:7627 NOTICE: SQLCODE: 1
Mar 15 20:42:47 pid:7627 NOTICE: SQL Error message: ERROR: duplicate key value violates unique constraint "idx_data_main2"; Error while executing the query
Mar 15 20:42:47 pid:7627 NOTICE: chlRegReplica cmlExecuteNoAnswerSql(insert) failure -809000
Mar 15 20:42:47 pid:7627 NOTICE: chlRegReplica cmlExecuteNoAnswerSql(rollback) succeeded
Mar 15 20:42:47 pid:7627 remote addresses: 10.15.0.6, 127.0.0.1 ERROR: [filePathRegRepl] - failed to register replica for [/tempZone/home/irodstester/hola], status:[-809000]
Mar 15 20:42:47 pid:7627 remote addresses: 10.15.0.6, 127.0.0.1 ERROR: [create_new_replica:362] - failed to register physical path [error_code=[-809000], path=[/tempZone/home/irodstester/hola], hierarchy=[noReplRoot;blue;ufs4]
Mar 15 20:42:48 pid:7627 remote addresses: 10.15.0.6, 127.0.0.1 ERROR: [replicate_data_object:748] - failed to replicate [/tempZone/home/irodstester/hola]
Mar 15 20:42:48 pid:7627 NOTICE: rsDataObjRepl - Failed to replicate data object. status:[-809000]
Mar 15 20:42:48 pid:7620 remote addresses: 10.15.0.6, 127.0.0.1 ERROR: rsExecRuleExpression : -809000, [-] /repos/irods_capability_storage_tiering/libirods_rule_engine_plugin-unified_storage_tiering.cpp:750:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) : status [CATALOG_ALREADY_HAS_ITEM_BY_THAT_NAME] errno [] -- message [iRODS Exception:
file: /repos/irods_capability_storage_tiering/libirods_rule_engine_plugin-unified_storage_tiering.cpp
function: void (anonymous namespace)::replicate_object_to_resource(rcComm_t *, const std::string &, const std::string &, const std::string &, const std::string &)
line: 110
code: -809000 (CATALOG_ALREADY_HAS_ITEM_BY_THAT_NAME)
message:
failed to migrate [/tempZone/home/irodstester/hola] to [noReplRoot]But all in all, it did migrate, and it did trim.
Note this was a single container - no networking weirdness, no 'wrong-host', no separate client machine.
trel
commented
10 months ago Sanger has now seen this behave on 4.2.11 and 4.2.12 in testing. Probably a candidate for closing...
korydraughn
commented
10 months ago Agreed. Please close.
After upgrading from 4.2.7 to 4.2.11 on Ubuntu 18.04, we found that the tiering plugin created a replica at the target tier, but did not remove the replicas from the source tier. This did trim correctly when on 4.2.7.
Packages;
metdata set;
server_config.jsonrule engine stanza;Note that the
roottree has a replication passthru, and so any objects in the tree should have two replicas, wheras the noReplRoot, as its name implies, has no such structureWe see the rules appear in the queue to migrate the objects once the metadata has been set on the object and it runs without obvious related errors in re, server or rodslog. Once the rules have run however we are left with three replicas, rather than one;