search

dmwm / T0

Apache License 2.0
7 stars 60 forks source link

Change the state of the request documents in WMStats after run is closed #3998

Closed dballesteros7 closed 11 years ago

dballesteros7 commented 12 years ago

For monitoring purposes, the Tier0Feeder should update the status of the request document for Express and Repack workflows through the update status call (updateRequestStatus in WMStatsWriter).

The transition should go from New to Closed. The monitoring will take care of mapping Closed to Repack and Express for repack and express workflows respectively.

@samircury can you take care of this one please?

samircury commented 12 years ago

Hi,

Given the discussions from yesterday, the state handling will be fully done on the Analytics component or we should still do something on the Feeder?

dballesteros7 commented 12 years ago

This one is still needed, no one but the Tier0Feeder knows about "a run is closed".

hufnagel commented 12 years ago

What you could do, in the table that we already use to track upload of the workflow request to couch, we just another flag to keep track of uploading the closing information. Then just run an additional query in the Tier0Feeder that checks all workflows that have been uploaded but not closed yet if their input fileset (run/stream or dataset related) has been closed and update the status in couch and t0ast when it has.

hufnagel commented 12 years ago

One question though, when should we update the status of the request document ? When all the data for Repack and Express in the Tier0 and the top level fileset is closed ? Or when Repack and Express are fully done ?

dballesteros7 commented 12 years ago

The first one, when all files are in the Tier-0. The second part is handled by the AnalyticsCollector based on the wmbs_subscription.finished flag

samircury commented 12 years ago

Diego is right, Dirk and I just concluded in the chat that we'll add one more flag to the workflow_monitoring page and use that . One question though, I need to make mechanisms to avoid reporting that for Reco, as it's something we want to Express/Repack only, right?

dballesteros7 commented 12 years ago

If it is too much work to avoid doing the same for Reco, it's okay. Reco can have the same New -> Closed transition, but in this case it really doesn't have any meaning as in the other two and the monitor will show Closed as PromptReco just as it would do with New.

samircury commented 12 years ago

Should be fine without Reco, if it gets too bad I complain here =)

samircury commented 12 years ago

I'm going to use the field name in workflow_monitoring as 'closeout', complain here if you don't like it.

samircury commented 12 years ago

This was actually a dumb question :-1: https://hypernews.cern.ch/HyperNews/CMS/get/wmDevelopment/442.html

I could just inner join with wmbs_fileset in the discovery query and bring the .closed flag... doing like that

samircury commented 12 years ago

@hufnagel , if you're happy, when you merge :

https://github.com/dmwm/T0/pull/4002

Please close this ticket.

Thanks, Samir

dballesteros7 commented 12 years ago

If you add "fixes #3998" in the commit message (not the description in GH, tried it already and didn't work), GH will close the referred issue when the pull request is merged

hufnagel commented 11 years ago

Fixed in #4002