JSON Output

[ffgalindo]

Add option to create JSON file(s) with the storage stats information according to the WLCG AccountingTaskForce format. https://twiki.cern.ch/twiki/bin/view/LCG/AccountingTaskForce

[ffgalindo]

Function has been set up using the "--json" option to output a json file with the storage stats information for the Dynafed site and for each of the endpoints following the WLCG format to a point. There are still questions how Dynafed should be reported, so for the time being it is more for the Dynafed admin to obtain a formatted summary.

[ffgalindo]

Closed as the output is there, if any changes are made another issue will be opened.

[ffgalindo]

The format is not correct and needs to be fixed/updated. Minimal example to follow, for ATLAS:

{ "storageservice": { "latestupdate": 1537785901, "name": "atlas-fed-fe1.triumf.ca", "storageshares": [ { "name": "ATLASDATADISK", "totalsize": 2089072092774400, "timestamp": 1537785901, "usedsize": 1821193950514079, "path": [ "/dynafed/atlas/atlasdatadisk" ] }, { "name": "ATLASSCRATCHDISK", "totalsize": 43980465111040, "timestamp": 1537785901, "usedsize": 19213512871092, "path": [ "/dynafed/atlas/atlasscratchdisk" ] } ] } }

[MarcusEbert]

The following is what DPM generates (example for Atlas and Belle in this case, from https://belle-dpm-01.na.infn.it/dpm/na.infn.it/home/belle/storagesummary.json ):

{
    "storageservice": {
        "datastores": [], 
        "latestupdate": 1568743202, 
        "name": "belle-dpm-01.na.infn.it", 
        "implementation": "DPM", 
        "storageendpoints": [
            {
                "qualitylevel": "production", 
                "interfacetype": "gsiftp", 
                "name": "gsiftp", 
                "assignedshares": [
                    "all"
                ], 
                "endpointurl": "gsiftp://belle-dpm-01.na.infn.it/"
            }, 
            {
                "qualitylevel": "production", 
                "interfacetype": "https", 
                "name": "https", 
                "assignedshares": [
                    "all"
                ], 
                "endpointurl": "https://belle-dpm-01.na.infn.it/"
            }, 
            {
                "qualitylevel": "production", 
                "interfacetype": "xrootd", 
                "name": "xrootd", 
                "assignedshares": [
                    "all"
                ], 
                "endpointurl": "root://belle-dpm-01.na.infn.it/"
            }, 
            {
                "qualitylevel": "production", 
                "interfacetype": "srm", 
                "name": "srm", 
                "assignedshares": [
                    "all"
                ], 
                "endpointurl": "srm://belle-dpm-01.na.infn.it:8446/srm/managerv2?SFN=/"
            }
        ], 
        "qualitylevel": "production", 
        "storageshares": [
            {
                "name": "BELLE", 
                "servingstate": "open", 
                "totalsize": 390326627860480, 
                "timestamp": 1568743202, 
                "assignedendpoints": [
                    "all"
                ], 
                "usedsize": 344016289828562, 
                "vos": [
                    "dteam", 
                    "belle/Role=production", 
                    "belle/Role=lcgadmin", 
                    "belle/desy", 
                    "belle", 
                    "belle/team", 
                    "belle/kek", 
                    "ops"
                ], 
                "path": [
                    "/dpm/na.infn.it/home/belle"
                ]
            }, 
            {
                "name": "Quota globale per tutte le VO", 
                "servingstate": "open", 
                "totalsize": 17592186044416, 
                "timestamp": 1568743202, 
                "assignedendpoints": [
                    "all"
                ], 
                "usedsize": 8672674620, 
                "vos": [
                    "nogroup", 
                    "atlas/lcg1", 
                    "ops/NGI/Croatia", 
                    "ops/NGI/Italy", 
                    "ops/Role=lcgadmin", 
                    "dteam", 
                    "ops/NGI", 
                    "dteam/cern", 
                    "ops/NGI/Greece", 
                    "atlas/it", 
                    "atlas/team", 
                    "dteam/NGI_CZ", 
                    "atlas", 
                    "ops"
                ], 
                "path": [
                    "/dpm/na.infn.it/home"
                ]
            }
        ], 
        "implementationversion": "1.13.1"
    }
}

[ffgalindo]

From Marcus:

I compared the dynafed_storagestats.json output with the Belle one and it seems the following things need to chance (not only the URL): storageendpoints/name is set to the endpoint behind dynafed but should be an identifier within the experiment endpoint url should be dynafed url interfacetype should be the dynafed interface (always https or http) assignedshares should probably be "all" instead of endpoint URL since all under this URL is used by the experiment? storageshares/assignedendpoints should probably also be "all" instead of the endpoint name

output-diff.txt