Closed amaltaro closed 3 years ago
The code in question is here: https://github.com/dmwm/das2go/blob/master/services/reqmgr.go#L151 and was suggested by Seangchan. The code needs to find reqMgr ids from given dataset name. They are required to search for proper config , see https://github.com/dmwm/das2go/blob/master/services/reqmgr.go#L236
So, since I queried various couchdb views for input and output datasets I need to know the replacement for all of them via reqmgr2 APIs. Please see all views I used: https://github.com/dmwm/das2go/blob/master/services/reqmgr.go#L151-L157 and provide me a replacement for all of them. Then I can change the code. Since the views return different data-structure then reqmgr2 the code should be adjusted accordingly.
Regarding sanitation, please open up separate request and I'll work on it, it looks like I missed this check for config API.
Regarding map of WMCore APIs, the das follows DAS maps for each service. The maps for reqmgr2 are defined here: https://github.com/dmwm/das2go/blob/master/maps/reqmgr2.yml and I only rely on /data/request API for reqmgr2 For configs I extract them from couchdb but this is done with code rather then with map due to non-direct logic involved if you have API replacement you need to tell me it.
Valentin, I would suggest you to completely remove these 2 urls: https://github.com/dmwm/das2go/blob/master/services/reqmgr.go#L155-L158
and replace the ReqMgr2 couch views here:
https://github.com/dmwm/das2go/blob/master/services/reqmgr.go#L151-L154
by the two REST API I provided above (/reqmgr2/data/request?....
)
IF, you only need the workflow id, then please add detail=False
to the query string. Otherwise - and I see you look up for the ConfigCacheID
- you need to fetch the whole document, not only the request id.
They are required to search for proper config , see https://github.com/dmwm/das2go/blob/master/services/reqmgr.go#L236
For these config cache id, we do not have a REST API yet. So you can keep that code around.
Here is the sanitization issue: https://github.com/dmwm/das2go/issues/28
Alan, I don't need workflow Id since it is not used to look-up configuration in couch. The couch call for configuration relies on ConfigCacheID, e.g./couchdb/reqmgr_config_cache/<ConfigCacheID>/configFile
, therefore there is no way for me to use detail=false. But obviously fetching the entire doc just to get a proper value is overkill. Therefore I suggest that you implement proper reqmgr2 API to return ConfigCacheID for given dataset. If I'll have this API I can remove the entire logic of parsing reqmgr/couch docs.
I'm afraid I won't be able to work on a new API in the coming couple of months. I'd suggest to proceed with the changes we have discussed so far, and in the future we can come up with a proper REST API to retrieve only the ConfigCacheID.
Note that the way those views are currently queried (include_docs=true
), you are anyways already fetching the whole documents.
I fixed both issues reported here and in #28 in DAS code, and I also opened up feature request in DMWM/WMCore to provide proper API for that, see https://github.com/dmwm/WMCore/issues/9699
The changes will be scheduled into next cmsweb upgrade once they pass validation.
Thanks, Valentin
@vkuznet Valentin, please let me know if this issue should get created somewhere else.
As mentioned in the WMCore meeting today, while scanning the CMSWEB frontend logs, I noticed the following 3 calls:
Searching for these couch views in WMCore, they have been deprecated 5 years ago (!): https://github.com/dmwm/WMCore/pull/5609
Can you please update them as follows:
requestByOutputDataset
to/reqmgr2/data/request?outputdataset=XXX
, e.g.: https://cmsweb.cern.ch/reqmgr2/data/request?outputdataset=/WplusToMuNu_NNPDF30_TuneEE5C_13TeV-powheg-herwigpp/RunIISummer15wmLHEGS-MCRUN2_71_V1-v1/GEN-SIM&detail=falserequestByInputDataset
to/reqmgr2/data/request?inputdataset=XXX
, e.g.: https://cmsweb.cern.ch/reqmgr2/data/request?inputdataset=/WplusToMuNu_NNPDF30_TuneEE5C_13TeV-powheg-herwigpp/RunIISummer15wmLHEGS-MCRUN2_71_V1-v1/GEN-SIM&detail=falseIn addition to that, can you please clarify which kind of request information you need? Is it just the workflow name? Or you need some other workflow meta-data. If it's the former, then please do always use
detail=False
.Last but not least, if you check the 3rd example, there is no data sanitization on DAS Server, which means, it will make a reqmgr2 call to whatever data input is provided by the user. In this case, it asks for workflows by output dataset, but it provides a LFN.
An extra request, would you have a map of all WMCore APIs used within DAS Server/client? We might have other such cases that were not spotted yet. Thanks