search

dmwm / DMWMMON

1 stars 8 forks source link

'level' depth should apply to LFNs, not PFNs #21

Closed nikmagini closed 9 years ago

nikmagini commented 9 years ago

Hi Natalia,

the storage dumps used for SpaceMon require PFNs instead of LFNs:

nikmagini commented 9 years ago

Hi Natalia,

the storage dumps used for SpaceMon require PFNs instead of LFNs:

https://twiki.cern.ch/twiki/bin/view/CMSPublic/SpaceMonSiteAdmin#Step_1_Locate_produce_working_to

Using PFNs makes sense, since we may want to use SpaceMon also for space outside the official CMS namespace.

However it has a significant drawback: the maximum allowed aggregation depth is 'level=6' which is applied at the PFN directory structure. Some sites already have 6 directories before reaching the CMS namespace! For example:

https://cmsweb.cern.ch/dmwmmon/datasvc/xml/storageusage?node=T2_FR_GRIF_LLR&level=6

This means that we don't get any detail below '/store', so in practice for this site we are only monitoring total space...

Maybe you can change the aggregation to start counting levels only when you find /store (using the same depth also for files outside the CMS namespace, e.g. if /store is at level 4 and you request level 6 you aggregate everything at level 6+4=10)?

nikmagini commented 9 years ago

The dir in the previous example was

/dpm/in2p3.fr/home/cms/trivcat/store

nataliaratnikova commented 9 years ago

Hi Nicolo, thanks for pointing this out, The client side tool is already counting levels starting from /store. The limit of 6 levels is implemented on the server side and only involves StorageUsage data service output. Good thing is that actual database contents contain all uploaded information for the deeper levels. I've doubled this limit on my development web server, and one can now see space usage under the /store for GRIF_LLR now: http://spacemon-dev.cern.ch/dmwmmon/datasvc/perl/storageusage?node=T2_FR_GRIF_LLR&level=8 [ Keep in mind that server is running on cern vm behind a firewall, so you may need to tunnel or wget it from lxplus] . I will next commit this quick fix to my private repo and make a pull request, so this can go to the next cmsweb release.

I am not sure why this hard limit was introduced. Performance-wise limit on the input record would make more sense. A few sites actually had to set lower numbers to reduce the record size, as they hit the server upload size limit. As mentioned at the meeting, I think we should have an API that would return a record exactly as uploaded by the sites. I will create a feature request assigned to myself :) .