search

goiwai / irods-decommissioning-2024

0 stars 0 forks source link

Unable to write to /KEK-T2K/ #1

Open hastingskek opened 3 weeks ago

hastingskek commented 3 weeks ago

Today we (T2K near detector data quality and calibration people) discovered that the automated jobs uploading data to the KEK iRODS have been failing since The evening of 2024-05-28. Eg.

[irodsdq@soffdata02 ~]$ iput test.txt
ERROR: putUtil: put error for /KEK-T2K/home/dataquality/data/test.txt, status =
-526013 status = -526013 UNIX_FILE_GET_FS_FREESPACE_ERR, Permission denied

T2K run 13 is scheduled to continue until the start of the next month. We had been under the impression that we could use iRODs until then.

goiwai commented 3 weeks ago

Let me propose two possible solutions:

  1. Roll back the zone KEK-T2K to access by iRODS client utilities by the end of Run 13 (approx. beginning of July 2024)
  2. Permit write access to specific user DN to an alternative access point for the zone: KEK-T2K

Please let me know which solution you and your team prefer.

hastingskek commented 3 weeks ago

We prefer option 1. since it seems option 2. would require extreme changes to automated workflows and as such is not appropriate for a very short term fix.

Please note that Run 13 in principle could continue until as late at July 8th and that we would want to be able still write data to iRODS until around 1 week after the run stops. So could you please keep it writable until at least July 15th?

goiwai commented 3 weeks ago

I have restored the ownership of the Zone KEK-T2K (hpssResc). I believe the data transfers run again as you did. I will keep the ownership until it is no longer necessary. If the issue remains, please let me know with a minimum code snippet so I can reproduce the problem. Good enough what you posted earlier.

Please note that the iRODS decommissioning and replacement with StoRM Storage Element is not a measure without consent. I had an agreement with T2K people about the matter a year ago.

NickHastings commented 3 weeks ago

I have restored the ownership of the Zone KEK-T2K (hpssResc). I believe the data transfers run again as you did. I will keep the ownership until it is no longer necessary. If the issue remains, please let me know with a minimum code snippet so I can reproduce the problem. Good enough what you posted earlier.

I can confirm that it is working now. Thanks for restoring it so quickly.

Please note that the iRODS decommissioning and replacement with StoRM Storage Element is not a measure without consent. I had an agreement with T2K people about the matter a year ago.

We were completely aware of the decommissioning but had (at least I had) been informed that the iRODS system would still be available for the T2K run prior to the J-PARC Summer maintenance period in 2024.

goiwai commented 3 weeks ago

I have restored the ownership of the Zone KEK-T2K (hpssResc). I believe the data transfers run again as you did. I will keep the ownership until it is no longer necessary. If the issue remains, please let me know with a minimum code snippet so I can reproduce the problem. Good enough what you posted earlier.

I can confirm that it is working now. Thanks for restoring it so quickly.

It's good to hear that. I would keep the issue open until read-only mode as KEK-T2K again.

Please note that the iRODS decommissioning and replacement with StoRM Storage Element is not a measure without consent. I had an agreement with T2K people about the matter a year ago.

We were completely aware of the decommissioning but had (at least I had) been informed that the iRODS system would still be available for the T2K run prior to the J-PARC Summer maintenance period in 2024.

You are perfectly correct. To be more precise, the iRODS system will keep running in read-only mode by the end of August 2024.