Open cfgamboa opened 2 years ago
Hi Carlos,
this is default value if no storage is configured in the target directory (not OSMTemplate and no sGroup tags)
For some reason the repository at the pool do not get the information written for the target directory. The replica does have the right TAGs
Hi Carlos, can you check the billing logs to find matching transfers to those files?
Hello @kofemann
There are two examples here 1. for 0000EF4CF609D39F4066958D3FCD6ADC4507 file
The billing records do not trace the change on the storage class in the pool
File stored
02.01 09:43:30 [door:GFTP-dcdoor18-186subnet-AAW6R2CULCg@gridftp-dcdoor18Domain:request] ["/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=atlpilo1/CN=614260/CN=Robot: ATLAS Pilot1":6435:31152:130.199.159.106] [0000EF4CF609D39F4066958D3FCD6ADC4507,27895219] [/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/48/fc/log.23744689._008438.job.log.tgz.1.rucio.upload] bnlt0d1:BNLT0D1@osm 743 0 {0:""}
02.01 09:43:30 [pool:dc051_10:transfer] [0000EF4CF609D39F4066958D3FCD6ADC4507,27895219] [/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/48/fc/log.23744689._008438.job.log.tgz.1.rucio.upload] bnlt0d1:BNLT0D1@osm 27895219 246 true {GFtp-2.0 10.42.38.51 38085} [door:GFTP-dcdoor18-186subnet-AAW6R2CULCg@gridftp-dcdoor18Domain:1612190609419000] {0:""}
**File replicated to other pool**
02.01 09:43:30 [pool:dc051_10:transfer] [0000EF4CF609D39F4066958D3FCD6ADC4507,27895219] [Unknown] bnlt0d1:BNLT0D1@osm 27895219 212 false {Http-1.1:10.42.38.94:0:dc211_2:dc211twoDomain:/0000EF4CF609D39F4066958D3FCD6ADC4507} [pool:dc211_2@dc211twoDomain] {0:""}
At the pool
[dcadmin02] (dc211_2@dc211twoDomain) admin > rep ls -l 0000EF4CF609D39F4066958D3FCD6ADC4507
0000EF4CF609D39F4066958D3FCD6ADC4507 <C-------X--L(0)[0]> 27895219 si={<Unknown>:<Unknown>}
[root@dc211 data]# stat 0000EF4CF609D39F4066958D3FCD6ADC4507
File: ‘0000EF4CF609D39F4066958D3FCD6ADC4507’
Size: 27895219 Blocks: 54488 IO Block: 4096 regular file
Device: 901h/2305d Inode: 17661544453 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-02-01 09:43:30.497048240 -0500
Modify: 2021-02-01 09:43:30.402044789 -0500
Change: 2021-02-01 09:43:30.402044789 -0500
2. This file 0000FF4DE24A73B14901A1F230DAF437CD1E had only one replica
File transfer
02.24 20:27:53 [door:GFTP-dcdoor11-15ipv6subnet-AAW8Hw1A2Pg@gridftp-dcdoor11Domain:request] ["/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ddmadmin/CN=531497/CN=Robot: ATLAS Data Management":6435:31152:2620:0:210:1:0:0:0:1b] [0000FF4DE24A73B14901A1F230DAF437CD1E,1000233694] [/pnfs/usatlas.bnl.gov/LOCALGROUPDISK/rucio/user/jhaley/dd/74/user.jhaley.24112049._000021.output_Loose_v1_MC16e_PFlow.root] GROUPDISK:LOCAL@osm 37790 0 {0:""}
02.24 20:27:53 [pool:dc046_3:transfer] [0000FF4DE24A73B14901A1F230DAF437CD1E,1000233694] [/pnfs/usatlas.bnl.gov/LOCALGROUPDISK/rucio/user/jhaley/dd/74/user.jhaley.24112049._000021.output_Loose_v1_MC16e_PFlow.root] GROUPDISK:LOCAL@osm 1000233694 35419 true {GFtp-2.0 10.42.38.64 34280} [door:GFTP-dcdoor11-15ipv6subnet-AAW8Hw1A2Pg@gridftp-dcdoor11Domain:1614216435981000] {0:""}
File transfer to other pool
02.24 20:27:57 [pool:dc046_3:transfer] [0000FF4DE24A73B14901A1F230DAF437CD1E,1000233694] [Unknown] GROUPDISK:LOCAL@osm 1000233694 3457 false {Http-1.1:10.42.38.111:0:dc225_3:dc225threeDomain:/0000FF4DE24A73B14901A1F230DAF437CD1E} [pool:dc225_3@dc225threeDomain] {0:""}
First record of the file with
03.28 00:29:19 [pool:dc225_3:transfer] [0000FF4DE24A73B14901A1F230DAF437CD1E,1000233694] [/pnfs/usatlas.bnl.gov/LOCALGROUPDISK/rucio/user/jhaley/dd/74/user.jhaley.24112049._000021.output_Loose_v1_MC16e_PFlow.root] <Unknown>:<Unknown>@osm 52562 725 false {Xrootd-5.0:10.42.38.67:58280} [door:Xrootd2-dcdoor18@xrootd2-dcdoor18Domain:AAW+kTRH9mA:1616905758376000] {0:""}
To fix the storage class the file is manually replicated to another pool, this seems to set the right Storage Class
09.27 10:09:03 [pool:dc225_3:transfer] [0000FF4DE24A73B14901A1F230DAF437CD1E,1000233694] [Unknown] <Unknown>:<Unknown>@osm 1000233694 5664 false {Http-1.1:10.42.64.32:0:dc255_1:dc255oneDomain:/0000FF4DE24A73B14901A1F230DAF437CD1E} [pool:dc255_1@dc255oneDomain] {0:""}
File with wrong class is later removed
09.27 10:51:32 [pool:dc225_3@dc225threeDomain:remove] [0000FF4DE24A73B14901A1F230DAF437CD1E,1000233694] [Unknown] <Unknown>:<Unknown>@osm {0:"'rep rm' command"}
Carlos
Is there any tool command that allows set the correspondent storage class
For example this file:
[dcadmin02] (dc255_1@dc255oneDomain) admin > \sl 000040178281650A426FA54E545CC05F4E96 rep ls 000040178281650A426FA54E545CC05F4E96
dc221_16:
000040178281650A426FA54E545CC05F4E96 <C-------X--L(0)[0]> 20278934 si={bnlt0d1:BNLT0D1}
dc252_4:
000040178281650A426FA54E545CC05F4E96 <C----------L(0)[0]> 20278934 si={<Unknown>:<Unknown>}
Replicating the file with si={
[dcadmin02] (dc255_1@dc255oneDomain) admin > \c dc252_1
[dcadmin02] (dc252_1@dc252oneDomain) admin > pp get file 000040178281650A426FA54E545CC05F4E96 dc252_4
Transfer Initiated
File SI is now displaying the si={bnlt0d1:BNLT0D1}
[dcadmin02] (dc252_1@dc252oneDomain) admin > rep ls -l 000040178281650A426FA54E545CC05F4E96
000040178281650A426FA54E545CC05F4E96 <C-------X--L(0)[0]> 20278934 si={bnlt0d1:BNLT0D1}
[dcadmin02] (dc252_1@dc252oneDomain) admin > \sn cacheinfoof 000040178281650A426FA54E545CC05F4E96
dc221_16 dc252_1 dc252_4
Migration move does not appear to fix the issue
[dcadmin02] (dc252_4@dc252fourDomain) admin > migration move -pnfsid=000040178281650A426FA54E545CC05F4E96 -target=pool dc251_1 -verify
[1] INITIALIZING migration move -pnfsid=000040178281650A426FA54E545CC05F4E96 -target=pool -verify -- dc251_1
File gets copied but the SI is stilling the one with
[dcadmin02] (dc252_4@dc252fourDomain) admin > \s dc251_1 rep ls -l 000040178281650A426FA54E545CC05F4E96
000040178281650A426FA54E545CC05F4E96 <C-------X--L(0)[0]> 20278934 si={<Unknown>:<Unknown>}
Carlos
Hi @cfgamboa ,
This means, that you have a directory without storage tags. The easiest way to find it out is to check the billing info for the given pnfsid.
Hello,
Thank you, the directory has a tag. The source file had the right one, it seems that the replica did get that attribute from the original one.
All the best, Carlos
On Jul 12, 2023, at 4:46 PM, Tiramisu Mokka @.***> wrote:
Hi @cfgamboa https://github.com/cfgamboa ,
This means, that you have a directory without storage tags. The easiest way to find it out is to check the billing info for the given pnfsid.
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6796#issuecomment-1633193332, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO3XM6XQIG2KPFJ6PZ3XP4EK5ANCNFSM6AAAAAAQW6UJZE. You are receiving this because you were mentioned.
Do you have corresponding billing records?
Yes, for the first example I have added the billing records. I could try to find more if you need that.
All the best, Carlos
On Jul 12, 2023, at 4:56 PM, Tiramisu Mokka @.***> wrote:
Do you have corresponding billing records?
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6796#issuecomment-1633204130, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMOYGC64FQ5TJPAWKIWTXP4FRRANCNFSM6AAAAAAQW6UJZE. You are receiving this because you were mentioned.
Dear all,
We have observed that via some of files on DISK have been marked as storage
si={<Unknown>:<Unknown>}
For example
Does anybody had reported similar issue? This does not appear to be isolated to a pool but rather scattered.
We are using dCache 7.2