search

DIRACGrid / DIRAC

DIRAC Grid
http://diracgrid.org
GNU General Public License v3.0
112 stars 174 forks source link

Site groups #219

Closed atsareg closed 12 years ago

atsareg commented 12 years ago

We have currently cases where jobs are eligible to run on more than one site but not on ANY site. This is typically the case for the jobs with input data. In this case the jobs while in the Waiting (Staging) status are having Multiple as the destination site in the Job DB. This value is appearing in the job monitoring and also in the WMS history accounting plots. This is not convenient. The proposal is to define Site groups, e.g. in /Resources/SiteGroups section, as lists of eligible site names and substitute the Multiple site description by the site group name whenever the list of sites for which the job is eligible coincides with the contents of the Site group. Banned sites should not taken into account by this logic as it will increase the number of possible site combinations.

Comments are welcome

graciani commented 12 years ago

Multiple is not just another SiteName, it is a key word that is used in the code to mark that the information in the Site Job Attribute is "fake" and one needs to go to the JDL to get the real list. Names matching those in the SiteGroups section will have to be handled in a similar way, but since the list is not necessarily constat, this is not simple and can lead to inconsistencies.

acasajus commented 12 years ago

That seems quite specific. I'd propose a simpler solution. Do not put anything into the Job Site until a site is actually defined for the Job. There should be another place where all the eligible sites should be. Special cases are always a bad idea and usually end up being anightmare to maintain.

phicharp commented 12 years ago

Comment to Ricardo's comment: the "Multiple" for going to the JDL could easily be avoided if we accept to go to the JDL even if there is one site in the attributes. This is AFAIK the only place where it is used. Therefore why not use this attribute for setting a meaningful representation of the set of sites? I sent a piece of cose to Andrei, just as an example...

As for the site groups, again I would not like to have to define the same things at two places: if one adds a Tier2 close to, say, IN2P3-RAW, it should be sufficient to add IN2P3-RAW as close SE to the site, without needing to define that site in the "CloudOf.IN2P3.fr" site group. In addition (as Andrei mentions) one of these sites may be banned in which case the list of sites is not the full list of the site group.

My proposal for the current operational problem is I agree a bit specific, i.e. to identify a set of sites by the Tier1 that is inside (if any). Otherwise, and less specific, we should use what is the real information, namely the targeted SE, and from that SE derive (in the CS for example) a "cloud" name. The TargetSE is in the JDL and could be used for that, right?

I don't see it but I had the memory that Ricardo proposed to add the SE location to the SE definition. This could be used for that purpose, e.g. if IN2P3-RAW is at LCG.IN2P3.fr, the site group for jobs targeting IN2P3-RAW could be CloudOf.in2P3.fr

acasajus commented 12 years ago

When the job site suddenly is not a real site, then there is something deeply wrong with the design. That sounds like special case on top of special cases just to see where the data is being staged/requested in the job monitoring. Maybe that should be in another page instead of trying to cram more stuff in the JM. Afaik Site has to be a real site. Changing that will lead to lots of unintended consequences like polluting the accounting.

Performance-wise, having to go to the JDL currently is really a huge penalty.

phicharp commented 12 years ago

I agree Adri, but the real site at which the files required by a job are being staged is LCG.IN2P3.fr, not "Multiple". I would have nothing against the site that appears when the file is staging is the real site name. When waiting, it can become Multiple, although, for reasons that are obvious to those who deal with operations, it would be highly preferable to have more information on the set of sites. For the time being, we are completely blind. Which useful information do you get from this plot: https://lhcb-web-dirac.cern.ch/DIRAC/LHCb-Production/lhcb_prod/systems/accountingPlots/WMSHistory#ds9:_plotNames12:NumberOfJobss9:_groupings4:Sites13:_timeSelectors2:-1s10:_startTimes10:2011-09-30s7:_Statuss7:Stagings9:_JobGroups26:00012503,00012504,00012518s9:_typeNames10:WMSHistorys10:_plotTitles26:Reprocessing:%20staging%20jobse and similar for waiting jobs?

Theoretical arguments are fine but here we are dealing with operational arguments: we don't have enough information...

phicharp commented 12 years ago

B.t.w is "Multiple" a site name ;-)?

acasajus commented 12 years ago

Until a site is not set for a job. Nothing should pop out in the site column. For staging, It'd be better if it's another page so the jobdb doesn't have more stuff in it. We've got lots of special cases thoughout the code and maintaining those increases the dev time exponentially. We should start cleaning. If something needs to change then so be it. That's what versions are for.

acasajus commented 12 years ago

Last phrase wasn't clear: I mean, as this is a new dev we don't need to wait to v7 to roll this out.

phicharp commented 12 years ago

We don't want to have to navigate between pages to know where a site is eligible. But the most important is the WMS history accounting: here we need to know at which site a job was waiting for files to be staged, then for which (group of) sites it was waiting, and finally at which site it is running.

acasajus commented 12 years ago

Regarding navigate between pages: I can be a page with all staging jobs together so It's not really a hard-to-do navigation.

Regarding WMS history: It's accounting NOT monitoring. WMSHistory is by far the most heavy type there. The accounting is not a plotting facility.

On 4 October 2011 17:43, Philippe Charpentier < reply@reply.github.com>wrote:

We don't want to have to navigate between pages to know where a site is eligible. But the most important is the WMS history accounting: here we need to know at which site a job was waiting for files to be staged, then for which (group of) sites it was waiting, and finally at which site it is running.

Reply to this email directly or view it on GitHub: https://github.com/DIRACGrid/DIRAC/issues/219#issuecomment-2287850

atsareg commented 12 years ago

Even if the issue is closed one last comment. The solution that I have coded in the #220 pull request is certainly a temporary one to help quickly the current operational issue. The solution is the following:

  1. After the staging site is determined by a random choice out of a list of possible sites ( current logic ), get the list of all the sites having common SEs with the chosen one. This operation is quite generic and not LHCb specific, I do not see a problem with that.
  2. Derive from the list of staging sites from 1. the group name. This is coded in a separate function and now it is LHCb specific because it relies on the Tier parameter in the site definition: if the group of sites is containing a T0 or T1 site, this site name is taken for the group name replacing the prefix by "Group". If not T1 site is found or there is no Tier parameters in the site definitions, the group name will stay "Multiple" as it is now. So, for non-LHCb installations nothing changes.
  3. The job Site attribute is assigned the group name from 2.

Both "Multiple" or "Group.XXXX.xx" Site values are not real sites as Adri points out but rather flags to say that there is more than one eligible sites but not all of them ( otherwise it is ANY site ). Both cases are interpreted the same way. May be it was better to do the group name as "Multiple.XXXX.xx" and tp check for the occasion of Multiple in the name to make the checks simpler for both cases. This is a detail. This logic was used like that for years now, so I do not introduce anything new. However, it helps in Monitoring ( first of all ) as a user can see where the job will go after staging. It will also help monitoring through accounting right now with WMSHistory plots. This last monitoring will dissappear soon ( I hope ) and its replcement will or will not be based on the job Site attribute, this we still have to decide.

Bottom line. The coded solution is least intrusive that I can think of, quick ( already in place ), generic as applicable both to LHCb and non-LHCb case, can evolve if we will decide to introduce a notion of Site groups, solves the operational problem of LHCb NOW. So, I think this is a reasonable compromise.

phicharp commented 12 years ago

Hi Andrei, Why can't we in a generic manner use the TargetSE that is set in the JDL (from the task I presume), and define the group from that. In the CS we can define the location of the SE (anyway we need to do that in order to know physically where it is, in case for example one wants to put the SE on a Google map ;-). Then if a job has a targetSE, it means it will be eligible to the "group" of that physical site. Once it is matched of course it is assigned to the actual site where it runs. The group is thus defined as "all sites close to that SE" My 2 cents, Philippe

Le 6 oct. 2011 12:34, Andrei Tsaregorodtsev a crit :

Even if the issue is closed one last comment. The solution that I have coded in the #220 pull request is certainly a temporary one to help quickly the current operational issue. The solution is the following:

  1. After the staging site is determined by a random choice out of a list of possible sites ( current logic ), get the list of all the sites having common SEs with the chosen one. This operation is quite generic and not LHCb specific, I do not see a problem with that.
  2. Derive from the list of staging sites from 1. the group name. This is coded in a separate function and now it is LHCb specific because it relies on the Tier parameter in the site definition: if the group of sites is containing a T0 or T1 site, this site name is taken for the group name replacing the prefix by "Group". If not T1 site is found or there is no Tier parameters in the site definitions, the group name will stay "Multiple" as it is now. So, for non-LHCb installations nothing changes.
  3. The job Site attribute is assigned the group name from 2.

Both "Multiple" or "Group.XXXX.xx" Site values are not real sites as Adri points out but rather flags to say that there is more than one eligible sites but not all of them ( otherwise it is ANY site ). Both cases are interpreted the same way. May be it was better to do the group name as "Multiple.XXXX.xx" and tp check for the occasion of Multiple in the name to make the checks simpler for both cases. This is a detail. This logic was used like that for years now, so I do not introduce anything new. However, it helps in Monitoring ( first of all ) as a user can see where the job will go after staging. It will also help monitoring through accounting right now with WMSHistory plots. This last monitoring will dissappear soon ( I hope ) and its replcement will or will not be based on the job Site attribute, this we still have to decide.

Bottom line. The coded solution is least intrusive that I can think of, quick ( already in place ), generic as applicable both to LHCb and non-LHCb case, can evolve if we will decide to introduce a notion of Site groups, solves the operational problem of LHCb NOW. So, I think this is a reasonable compromise.

Reply to this email directly or view it on GitHub: https://github.com/DIRACGrid/DIRAC/issues/219#issuecomment-2308451

Dr Philippe Charpentier J09210, Physics Department, CERN, CH-1211 Genve 23 LHCb Experiment Distributed Computing Coordinator Office 2-R-007, Tel : +41 22 767 4244 , GSM : +41 76 487 0167 Mailto:Philippe.Charpentier@cern.ch http://cern.ch/Philippe.Charpentier

atsareg commented 12 years ago

It is a possibility. In a longer term however, the notion of a close SE will be blurred as it is already happening now. Krakow is not closer to GridKa than to CERN, for example, it is rather an operational convenience. So, grouping sites by closeness to a SE is something ill-defined. On the other hand if even the group composition changes it can still stay as "Group of sites considered close to the SE lambda according to the current operations policy". The site grouping will then be dependent on the operation practices of a given team. Then it should be rather implemented as a VO specific plugin that can be configured and called from the framework. What is coded now is giving a similar result to what you are proposing. However, this should be seen as a temporary solution ( rather ugly ) and a better one should be seeked. Plugin can be handy if no consensus solution will be found.