irods / irods_capability_storage_tiering

BSD 3-Clause "New" or "Revised" License
5 stars 10 forks source link

Empty data object transfer between storage tiers #266

Open ll4strw opened 4 months ago

ll4strw commented 4 months ago

I have two resources with the following settings and a rule that executes periodically the tiering plugin

$ imeta ls -R laptopResc  
AVUs defined for resource laptopResc:
attribute: irods::storage_tiering::group
value: groupie
units: 0
----
attribute: irods::storage_tiering::time
value: 15
units:

$ imeta ls -R laptopResc1
AVUs defined for resource laptopResc1:
attribute: irods::storage_tiering::group
value: groupie
units: 1

Upon itouching an empty data object into laptopResc, this file stays there forever and does not get moved to laptopResc1. On the other hand, any other data object, which contains data, honors the tiering rules

$ itouch test
$ itouch test33
$ echo "hello" >> test1 && iput test1
$ echo "hello" >> test2 && iput test2
$ ils -l
/laptopZone/home/rods:
  rods              0 laptopResc            0 2024-03-20.09:29 & test
  rods              1 laptopResc1           11 2024-03-20.10:00 & test1
  rods              1 laptopResc1           11 2024-03-20.10:09 & test2
  rods              0 laptopResc            0 2024-03-20.10:10 & test33

However, if I have the following local directory made of an empty file (test33) and a data file (file1)

$ ls -l TEST33
total 4
-rw-rw-r-- 1 irods irods 5 Mar 20 10:15 file1
-rw-rw-r-- 1 irods irods 0 Mar 20 10:15 test333

and I copy it over to iRODS

irsync -R laptopResc -r TEST33 i:TEST33

then the empty file DOES get moved to the second storage tier

$ ils -l /laptopZone/home/rods/TEST33
/laptopZone/home/rods/TEST33:
  rods              1 laptopResc1            5 2024-03-20.10:18 & file1
  rods              1 laptopResc1            0 2024-03-20.10:18 & test333

Finally, by adding again an empty data object to the TEST33 collection results in a missed transfer to the socond storage tier again

$ itouch TEST33/empty_file
$ ils -l /laptopZone/home/rods/TEST33
/laptopZone/home/rods/TEST33:
  rods              0 laptopResc            0 2024-03-20.10:22 & empty_file
  rods              1 laptopResc1            5 2024-03-20.10:18 & file1
  rods              1 laptopResc1            0 2024-03-20.10:18 & test333

Is this the expected behavior? If so, why to treat differently an empty file generated using itouch and one generated via irsync? Thanks in advance, L.

trel commented 4 months ago

Definitely not expected behavior.

Which version of iRODS is this?

ll4strw commented 4 months ago
Ubuntu 22.04.3 LTS

irods-server/jammy,now 4.3.1-0~jammy amd64
irods-icommands/jammy,now 4.3.1-0~jammy amd64
irods-rule-engine-plugin-unified-storage-tiering/jammy,now 4.3.1.1-0~jammy amd64
trel commented 4 months ago

Got it - we'll work to reproduce what you're seeing.

korydraughn commented 4 months ago

The plugin does not respond to itouch because the plugin doesn't know about the PEPs for the touch API.

We'll get that resolved.

alanking commented 4 months ago

Subset of https://github.com/irods/irods_capability_storage_tiering/issues/200, I believe (as seen here: https://github.com/irods/irods_capability_storage_tiering/issues/200#issuecomment-1112508340)

trel commented 4 months ago

ah, indeed.