DAS combined results with RUCIO group scopes

[dciangot]

From the discussion on importing tua embedded samples into Rucio (*), we are now left with one last point to solve. We need to understand IF and HOW we want to allow DAS to "discover" RUCIO information for group scopes (and later maybe also user ones). If I navigated the code correctly, what I see from here (**) is that the (correct so far) assumption is for any block or file to be under the cms scope. In this case we have this samples published as /EmbeddingRun2016_H/MuonEmbedding-inputDoubleMu_106X_ULegacy_miniAOD-v1/USER in phys03 and in rucio with the container name group.pog_tau_group:/EmbeddingRun2016-HIPM_B_ver1/ElectronEmbedding-inputDoubleMu_106X_ULegacy_miniAOD-v1/USER.

I could help to produce a patch for this (together with @dynamic-entropy) in case we agreed that this kind of discovery is something that we want.

@vkuznet @ericvaandering @belforte and others, what do you think?

(*) https://cms-talk.web.cern.ch/t/moving-cms-embedding-samples-run-2-ul-to-t1s-global-storage-area/10203 (**) https://github.com/dmwm/das2go/blob/master/services/combined.go#L223-L427

[belforte]

I think that we need a "policy" discussion on this.

Diego btw, if someone were to produce a patch for DAS, how will the correct scope be obtained, since the prog_tau_group is not a part of the dataset name ? Do you suggest to add to DAS/dasgoclient an option to specify the Rucio DID scope ?

I do not like the idea of using group scopes like this, opening up to having "CMS datasets" fragmented in a lot of small independent sets. One thing is user datasets as produced by e.g. CRAB where username will be carried around forever in the dataset name. The other is this sample which IIUC we want to import in CMS au-pair with P&R data, host on tape, use in input to future reprocessing etc. Are there technical reasons why LFN's need to be in group scope, or was it a deliberate choice to setup a policy ?

If anything, we may need to revisit how we name datasets used in publication from CRAB to make sure that Rucio scope can be obtained algorithmically from the DS name.

[belforte]

or maybe this is simply a fake problem ? The dataset can be found w/o problems in DAS at https://cmsweb.cern.ch/das/request?view=list&limit=50&instance=prod%2Fphys03&input=%2FEmbeddingRun2016_H%2FMuonEmbedding-inputDoubleMu_106X_ULegacy_miniAOD-v1%2FUSER if Artur is simply still in the process of moving things to prod/global ? When that happens (whether done by him or by P&R) DIDs will have to be moved to cms: scope and everything will be fully consistent. No ?

What exact procedure do we want here ? I surely lost a bit track of things, I thought it was a "once ever" case in which we elevate a user dataset to same status as official production.

[dciangot]

No, wait, the problem is less critical than what you thought.

In fact, as you see, the dataset is perfectly published. The problem is ONLY to make das aware of the site availability via Rucio.

[vkuznet]

I would agree with Stefano, first CMS needs to understand if difference Rucio scopes will be a norm. If so, then in DAS you need to introduce a special keyword for it, like instance for DBS instances. If this is the case then I outlined procedure of what will be required on DAS side:

• discussion and rules about Rucio scopes and how to handle them
• Rucio team should clarify how to properly construct Rucio URLs using different scopes we need to decide how to specify this scope in DAS, e.g. we may introduce a special scope DAS keyword which user will need to specify.
• once decision has been made somebody need to implement it which will involve:
  • add new keyword into list of DAS known keys
  • change DAS server and DAS CLI to deal with new keyword
  • deploy DAS server to k8s
  • create new DAS dasgoclient and upload it to CMSSW

Such that at the client user will specify:

dasgoclient -query="site dataset=/a/b/c scope=pop_tau instance=prod/phys03"

or in web UI, we'll need to add additional drop-down to specify known Rucio scopes similar to DBS instances.

Bottom line, to start the process you first need to identify what is correct Rucio URL to query, e.g.

https://cms-rucio.cern.ch//replicas/pog_tau//EmbeddingRun2016_H/MuonEmbedding-inputDoubleMu_106X_ULegacy_miniAOD-v1/USER#a5ef5cee-1b07-4a50-b6af-87e0c99ba94e/datasets
or
https://cms-rucio.cern.ch//replicas/group:pog_tau//EmbeddingRun2016_H/MuonEmbedding-inputDoubleMu_106X_ULegacy_miniAOD-v1/USER#a5ef5cee-1b07-4a50-b6af-87e0c99ba94e/datasets
or
https://cms-rucio.cern.ch//replicas/group.pog_tau/_group/EmbeddingRun2016_H/MuonEmbedding-inputDoubleMu_106X_ULegacy_miniAOD-v1/USER#a5ef5cee-1b07-4a50-b6af-87e0c99ba94e/datasets

see, part of the path right after replicas. So far I have no clue what is correct Rucio scope for this publication. Once you know the URL, I suggest that you first query it via curl to get some data, and once you'll get data we can decide how to implement it in DAS.

Once again, you should consult with @klannon about human effort to implement all of this as my time is no longer allocated for DAS development.

[belforte]

I hope things can be solved w/o any change in DAS. If not, clearly we need to involve whoever is maintaining DBS and DAS now. IIRC there is a person at FNAL with 25% FTE on DBS, I do not remember the name, did he disappear ? Do we have anybody on DAS ? It works perfecly now and maybe we do not need to do any change, but... still, if there's nobody behind it, we better deprecate/shutdown and stop wasting Valentin's time with these questions.

[belforte]

The problem is ONLY to make das aware of the site availability via Rucio.

I offer a simple solution: do not use DAS for location of data not in CMS scope.

[dciangot]

That would be my favorite one as well.

[ArturAkh]

Hi Artur here,

In that case: what would be the alternative tool for (non-power-)users? As far as I remember, as a student, the first site/tool for users was DAS, and not DBS or Rucio. Please consider this not only from the expert perspective, but also from user perspective.

All in all, I think it would be good to use the same tool for all datasets - which can be different from DAS. What would be then the alternative? I assume rucio or DBS separately?

[ArturAkh]

And here the info on which scope to use with Rucio:

arturgo@arturgo-ThinkPad-P14s-Gen-2i:~/HappyFace/HappyFaceCore/dev-scripts$ rucio list-dids group.pog_tau_group:/EmbeddingRun2016_H/MuonEmbedding-inputDoubleMu_106X_ULegacy_miniAOD-v1/USER
/home/arturgo/.local/lib/python3.10/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.8) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
+--------------------------------------------------------------------------------------------------+-------------------+
| SCOPE:NAME                                                                                       | [DID TYPE]        |
|--------------------------------------------------------------------------------------------------+-------------------|
| group.pog_tau_group:/EmbeddingRun2016_H/MuonEmbedding-inputDoubleMu_106X_ULegacy_miniAOD-v1/USER | DIDType.CONTAINER |
+--------------------------------------------------------------------------------------------------+-------------------+

[belforte]

the proper question is: why does Artur need to use DAS/dasgoclient here, at this point in the process of moving-cms-embedding-samples-run-2-ul-to-t1s-global-storage-area ?

BYW, at the moment AFAICT DAS (and CRAB) report the data location from the DBS origin_site field as T1_DE_KIT_Disk [1], so people should be able to access the data. @ArturAkh which exact problem do you have ?

[1] https://cmsweb.cern.ch/das/request?instance=prod/phys03&input=site+dataset%3D%2FEmbeddingRun2016_H%2FMuonEmbedding-inputDoubleMu_106X_ULegacy_miniAOD-v1%2FUSER

[belforte]

I understand that if dataset is replicated elsewhere than KIT, current setup will not track it. That is known since ever and when we will support CRAB output being put in /store/user/rucio AND published in DBS/prod/phys03 we will also add in CRAB to locate those data using Rucio in the user scope.

If and when peple will use /store/user/rucio for storage and Rucio for ASO in CRAB, .. well, they better start getting familiar with Rucio and be able to track replicas and rules there directly. We will provide documentation and maybe some utility scripts. But extending DAS scope does not look a good solution to me.

[ArturAkh]

From technical point of view, there is no problem, if DAS is not doing anything actively, and there are no dependencies on DAS in terms of running jobs (with WM tools, crab, etc.), in terms of replicating the datasets to other sites, where users can access them in a better way.

If this is all fulfilled, then OK.

However, what I'd like to stress out, is that if the mostly used tool/service for usual users is DAS (and not DBS + Rucio), then an update of DAS is required, too, on my opinion - to ensure user-friendliness. If there isn't any more appropriate person power to resolve that, I'm happy to jump in...

[ArturAkh]

f and when peple will use /store/user/rucio for storage and Rucio for ASO in CRAB, .. well, they better start getting familiar with Rucio and be able to track replicas and rules there directly. We will provide documentation and maybe some utility scripts. But extending DAS scope does not look a good solution to me.

In that case, we should not advertise DAS + dasgoclient as the tool for nagivating and exploring dataset information anymore, but let students and users learn directly DBS + rucio (which is more complicated on my opinion, btw).

[vkuznet]

Let me give you my view on this situation:

• the CMS is very complex system and users may have little knowledge in its internals
• said that, DBS or Rucio setup is not trivial
• DBS and Rucio tools are different and have different APIs which you'll need to teach end-users
• DAS was designed to simplify things, such as
  • it provides clean query syntax to complex DBS/Rucio (and other APIs) in uniform way via DAS queries
  • it is static executable which does not require any setup, just grab and use it

If you want to write wikies, docs, about DBS/Rucio tools that's fine but I think you'll lost your users in this complexity since users will need to know when and which tool to use. IN some cases it will be DAS, in others DBS or Rucio or their combination. In my view, it is not a good option either.

I suggest that you evaluate this use-case properly and decide will CMS in a future use different Rucio scopes, if so please make an effort to fix DAS to do that and keep users using simple tool.

[ArturAkh]

:+1:

Totally agreed :) And that's why I'm more in favour of adapting DAS (at least as of now).

And I'm sorry for being provocative with my statements, but - as you also say @vkuznet - it is very important to think about the point, that currently DAS + dasgoclient are the tools to go for end-users, especially for master students and PhD students as non-experts, who have to do their physics analyses and service work in a limited amount of time.

But yes, this should be discussed and decided within CMS in a wider audience.

If we all think, it would be good to extend DAS due to the fact, that group scopes are to be used in a more frequent way in the future, then I'd be happy to jump in for developments, if there isn't any other more suitable person at hand.

[belforte]

as Valentin said, first thing is to understand where exactly we want to use multiple Rucio scopes and for what. And keep things simple. Already global vs. phys03 is a source of confusion.

Surely DAS helps, I use it all of the times !

[ericvaandering]

I have to agree with this e-mail.

It is true that the production tools (CRAB, WMAgent, etc) do no use DAS. Only users use DAS. However DAS has been so successful that most of our users no longer realize DBS even exists anymore, they assume DAS is keeping track of dataset information.

So it seems to me DAS should be able to give users the information they want (dataset structure AND location(s) plus all the lumi info in DBS). If they are power users enough to get this from multiple sources, great. But they shouldn’t have to.

Eric

On Jan 23, 2023, at 6:37 AM, Artur Gottmann @.***> wrote:

From technical point of view, there is no problem, if DAS is not doing anything actively, and there are no dependencies on DAS in terms of running jobs (with WM tools, crab, etc.), in terms of replicating the datasets to other sites, where users can access them in a better way.

If this is all fulfilled, then OK.

However, what I'd like to stress out, is that if the mostly used tool/service for usual users is DAS (and not DBS + Rucio), then an update of DAS is required, too, on my opinion - to ensure user-friendliness. If there isn't any more appropriate person power to resolve that, I'm happy to jump in...

— Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_das2go_issues_52-23issuecomment-2D1400274696&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=FIJoq-YFlCAT480CM6WwZ-anLDNk4MI3htwu-Wd76yQhGrU2eT_9XHCvS7SVkrM9&s=aiR0CzpYbUPihrWEgvouswTnq590Lg0oCTnuP_ntrYs&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAMYJLT3XNR7EEA7IBCTBQLWTZ3RJANCNFSM6AAAAAAUDSIGFA&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=FIJoq-YFlCAT480CM6WwZ-anLDNk4MI3htwu-Wd76yQhGrU2eT_9XHCvS7SVkrM9&s=eu1mCYwvrlobSmP7g273RacrSfrLblFs-Y8bOe0aecI&e=. You are receiving this because you were mentioned.

[belforte]

ahem... how can we move on from personal views to a clear policy/design to implement (with attached manpower of course!) ? Do we really want to support datasets in scopes other than cms ? In particular scopes that can't be derived from the dataset name ?

[ArturAkh]

Maybe I'm misunderstanding you here, @belforte - but I disagree, that I'm presenting only my personal view here.

I'm not only speaking for myself, but also for a fraction of CMS users. Including the ones from KIT, but - maybe - also from other institutes. In case of KIT, it is even my job to represent them in such discussions.

And as far as I can tell, it would be much in favour of these non-expert users, if they can stay with using DAS as a tool - also in case of CMS datasets with scopes other than cms (if DAS remains the state-of-the-art tool for end-users).

[ericvaandering]

ahem... how can we move on from personal views to a clear policy/design to implement (with attached manpower of course!) ? Do we really want to support datasets in scopes other than cms ? In particular scopes that can't be derived from the dataset name ?

There was a long process leading to this point going all the way up to our L1s where it was explicitly stated that this would be a first test of a process like this and would go into a group scope. Maybe we made a mistake, but it wasn't a snap decision at any point

[belforte]

My apologies @ericvaandering and @ArturAkh I did not mean to imply that the previous process was flawed or that I do not understand the point about keeping it simple for users. I was only referring to this specific thread where I do no see the process by which a decision can be made. Be it to move data scope or to upgrade DAS or whatever.

We surely owe a lot now to Artur's willingness to spearhead this topic and his determination in overcoming all difficulties, indeed it is very good to be talking in front of a concrete example and concrete problems. Maybe we should look at the most expedient way to limp along and reach the end of the process so that we experience the full thing.