Weighting

[ESPD]

For some selection criteria it is important that the public buyer can define how the criterion is weighted. We need define what are the attributes to achieve this.

[rantati7]

at least in Finland weighting comes to play only if selection criteria is used for reducing the number of candidates in two phase procedures i.e. restricted, competitive procedure with negotiation or competitive dialogue. Maybe also in innovation partnership (at some phase) but I do not yet understand the procedure well enough to say anything affirmative.

[rantati7]

Weighting comes to question in Part V: Reduction of the number of qualified candidates.

Currently the question is only shown to EO. It should first be "configured" by CA and only then answered by EO.

For the commission ESPD service it would still be easier if parts IV and V would not be part of ESPD. These parts are linked with the object of procurement. All parts related to object should be built inside eTendering systems - not in Commission ESPD service! Just an opinion that I have had when I first time saw the draft of ESPD regulation.

[lauramrtn]

Currently within Scotland, scoring and weighting is used for a two stage procedure to restrict the number of economic operators that you take through to the next stage.

The most suitable place for weighting I believe is part IV C - Technical and Professional Ability, as the other sections are more clearly pass/fail. I have attached a very basic (non-technical) idea of how I see this working but happy to provide more detail/explanation if this would be useful. Of course, any scoring/weighting methodology would need to be explained to the supplier in the contract notice.

Scoring Example - ESPD July 2016.docx

[rantati7]

Similar method is used in Finland for the same purpose.

We are currently doing the scoring inside our eTendering system but when ESPD is build-in we need to enhance it with the scoring. The tricky question is how should the need to enhance ESPD (in national eTendering systems) be reflected in the Data Model.

We need to enhance ESPD also in other parts - namely by inputting minimum requirements to e.g. turnover and the number of years the turnover is required e.g. 2015, 2014 and 2013 (see the other issues I have opened in Github).

[ec-mcs]

Comments during the ESPD EDM eMeeting on 13.09.2016

Rudolf: The key is that the CA can evaluate on solid grounds. It will be a challange to decide, e.g. between "poor" and "acceptable" . The ciriteria for reaching a certain score must be very clear. Otherwise the CA will have a lot of legal troubles.

Steve Patterson: agreed - the scoring and weighting methodolgy must be made clear to EOs up front

Steve Patterson: it is much better to see it agains tthe question itself

Steve Patterson: in the request document only yhe question weighting and scoring methodology would be vsisble to bidders

Steve Patterson: yes, clearly hat actual score cannot be calvculated until al lof the ESPD's are returned and evaluated.

Rudolf: No, in the invitation to submitt a tender.

Steve Patterson: yes, we would inform them after the evaluation

Steve Patterson: the scoring and weighting is only relevant in a two stage procedure

Rudolf: Exactly!

Steve Patterson: yes, that's right

Steve Patterson: Ther system wold them multiply the actul score by the question weighting tto give the weighted score

Steve Patterson: I'm assumingthe buer would only score and weight questions thatare relevant to their particlulat procurement exercise

Steve Patterson: yes, MC that sounds right

Rudolf: Weighting criteria should clearly be distinguished /separated from the other criteria

Steve Patterson: agreed -mostother criteria are based on meeeri=ing minimum standatds are would be pass/fail

Steve Patterson: sorry, my typing is appalling!

Rudolf: Yes, we call those criteria concerning suitability K.O. criteria. The weighting criteria (selection criteriea) chose between the suitable EO.

Steve Patterson: even technicsl and professionali ability could be pass/fail in a single stage procedure, but the CA would need ot clearly define the minumum standat=rds required within the self-contained ESPD

Steve Patterson: Yes, I think so, althouh in practice I think think this is one aspect of ESPD CA's are finding most difficult

Rudolf: Minimum standard: E.g. the EO must employ a minimum of three technicians of a certain field or has to have certain certificats, e.g. ISO.

Jose Luis Cueva: I believe text fields should be avoided as much as possible, because of language problems

Steve Patterson: I think it's relativly easy when referring the things like ISO,ut much more difficult for questins relatinging to an EO's experience

Jose Luis Cueva: could we use "quantity" and "unit of measure"?

Steve Patterson: These are diificult questions for a CA to kdefinewhat a good answer woild be up-front

Jose Luis Cueva: the "unit of measure" is dependent of the criteria

Steve Patterson: I think most CA's in Scotlkand are not using the manpower question, simply because the CA cannot define in advance what "good" looks like

Steve Patterson: For us, the experience question is by far the most important in terms of scoring/weighting

Rudolf: The experience can be shown omostly by acomplished contracts of the past.

Steve Patterson: Rudolph, I agree. It is then up to the CA evaluators to decide and score how relavant the examples provided are to the current requirement.

Jerry Dimitriou: It is likely that the criteria using weighting will part of the national criteria and will not be part of the official EU criteria

Jose Luis Cueva: ok

Andrew Frost: ok

Jerry Dimitriou: Tuesday, right?

Peter van de Kletersteeg: ok

Steve Patterson: Jerry, I think the scoring and weighting could potentilay be applied to the entire technical and professional ability section, in a two-stage prrocedure

Jerry Dimitriou: @Steve, the problem is not weighting but the change on the requirement/question structure of the ESPD

[paulakeen]

Hi all,

We herein attach a figure illustrating a possible proposal to be discussed in our next meeting. In principle the solution shouldn't be much more complicated than that: two new fields have been added to the Criterion class, at the same level than FulfillmentIndicator element (which is the equivalent to the pass/fail indicator we were discussing during the previous meeting), (1) "Score" (normally an integer, but it could be a decimal), and (2) "Weight" (a percentage).

Also to discuss next meeting would be possible business rules associated to the weight, such as: (1) if a pass/fail criterion is specified in the XML instance no weight nor score would be allowed (or vice-versa); an exception should be thrown; (2) if the procurement project does not belong to a two-staged procedure (e.g. restricted, competitive dialogue, etc.) no score and weight is expected and an alert should be raised (but the XML probably shouldn't be rejected); (3) other....please think about this and put forward your proposals.

If this approach is accepted we'll provide a new versioned XSD and XML example instances.

KR

[paulakeen]

I've been reviewing the proposal put forward by UBL-2.2 and it's simpler and would make more sense (see image attached).

In this proposal there is no placeholder for the score and the weight is a code instead of a percentage. My gess is that they do not model the score as it belong to the evalutaion, which is out of the scope of the ESPD (they name it "QualificationApplicationResponse") and, most importantly, it would imply modifying the response of the EO to add the score which would tamper with the response understood as a record. As for the weight, different systems beyond a mere percentage can be used to weigh the criterion response, therefore a code can cope with a high variety of weighting solutions. The problem with this approach would be coming up with a sufficiently standard code list at the EU level.

[ec-mcs]

Here is the picture depicted on how the information will be shown to CAs/EOs

[ec-mcs]

Here is the scoring example from Scotland Scoring.Example.-.ESPD.July.2016.docx

[paulakeen]

Following the discussion held during the last workshop i herein paste a new proposal for the weight. First of all some general considerations:

1. Remember that for the Release v1.1.0 we're proposing not to use the ESPDRequest and work only with the ESPDResponse (therefore when speaking of the ESPD Schema we'll be referring always to the ESPDResponse);
2. We said that, in order to implement both the "transparency" and the "self-containment" principles, we need to allow the CA to specify the methodology used for weigthing;
3. We also said that we should provide a system to distinguish when a criterion is "pass/fail" or a "weighted" one;

The figure below provides a simplified view of the current ESPDResponse structure:

To implement the above requests:

1- A new element "WeightScoringMethodologyDescription" is added to the ESDPResponse root element. The figure below illustrates this:

2- The criterion element is extended with three new elements:

• EvaluationMethodTypeCode: a code specifying whether the criterion is of type fail/pass, weigh in, or may be another one (although not probable the suggestion is not to control this via an indicator).
• Weight: a code representing possibly a percentage, a formula returning a weight or whatever other solution specifying a relative weight the CA may come up with
• WeighingConsiderations: a text file to describe any particular circumstance around the weighing of the criterion or a specific aspect of the application of the general weighing methodology to this particular criterion.

[ESPD]

Example

EvaluationMethodType = Weight Weight = 10% (Percentage/Points) WeightingConsideration 0% 0 IT specialists 50% 1 IT specialists 100% 2 IT specialists

Weight = 60Points WeightingConsideration 0Points 0 IT specialists 30Points 1 IT specialists 60Points 2 IT specialists

[ESPD]

Chat notes in Adobe Marc-Christopher SCHMIDT: Hello everyone. We will start at 13:04

Fani TSIRANTONAKI: I am here :-)

Steve Patterson (Scottish Government): Thanks. I don't pretend to understand the coding aspcect, but what you have described seems good to me.

Rudolf: Does this mean, that the CA is putting down the rules for the weighting in the tender documents and the EO responses to this rules in the ESPD response?

Rudolf: YES!

Lois Devey 2: yes

Laura Martin: Yes, this seems to meet our needs.

Rudolf: Example:e-tendering for an e-procurement platformCriteria I: Key team1 IT specialist = 0 points2 IT specialists = 30 points3 IT specialists = 60 pointsCriteria II: Procurement procedures per year over the last three years1000 procedures with the platform = 0 points2000 procedures with the platform = 50 points3000 procedures with the platform = 100 pointsCriteria III: Main customers – minimum license fee for the platform in use € 100.0002 customers = 0 points5 customers = 40 points8 customers = 80 points

Laura Martin: Yeah, of course. We will review and add any other suggestions to Github.

Rudolf: This example is only based on points, no percentage.

Maria Font: @Rudolf as the weights are codified the fact that they are points instead of percentages doesn't really matters

Rudolf: In trhis example on the total of points is used for the ranking of the applicants. Based on this the best three or five will be choosen. That means ther should be also the possibility just to use points. without percentages

Maria Font: @MC, @Rudolf +1

Rudolf: The weighting should be optional for the CA to put into the ESPD.

Maria Font: It is optional in the schema

Rudolf: If is an open proceedure even the option of weighting should not be shown to the CA.

Laura Martin: Yes, we agree.

Rudolf: YES!

[lauramrtn]

Hi MC

We have reviewed the 3 new titles on scoring and weighting and think the firs two are fine (Evaluation Type Code and Weight) but we think out buyers might not understand weighting considerations. Would it be possible to rename it to something like 'Evaluation Information'. This would mean it would also be suitable where the question is pass fail and not just for those weighted. Not sure if anyone else has any thoughts on this or any other suggestions?

[mc]

this sounds fun to evaluate, but you will have to give me more time to understand the full complexity.

-mc (the github user you're notifying when using @mc, when indeed you probably meant to notify @ec-mcs)

[ec-mcs]

@mc sorry for this. I will ask @paulakeen not to use this anymore ;-)

[ec-mcs]

@lauramrtn, @paulakeen I would suggest that we use the following naming: EvaluationCode EvaluationWeight EvaluationInformation

Or alternatively WeightCode Weight WeightInformation

btw, it is only how we will call this in the datamodel, the user interface could show a different label

[lauramrtn]

@ec-mcs @paulakeen I like the first suggestion:

EvaluationCode EvaluationWeight EvaluationInformation

[paulakeen]

@ec-mcs @lauramrtn the UBL good-practice would prefer these other:

EvaluationType (or even EvaluationTypeCode) EvaluationWeightType (or even EvaluationWeightTypeCode) EvalutionInformation

Please could you help in providing a definition for each one of these elements? *tks

[Zilvaras]

Please be informed that during restricted procedure in Lithuania we calculate the score of each weighted qualification criteria by comparing it with the same qualification criteria of other EO.

For example:

Weighted qualification criteria (max weiht for all criteria 10 points):

1. Number of similar contracts (K1), weight of the criteria (L1) - 5 .
2. Turnover (K2), weight of the criteria (L2) - 3.
3. Number of emploees (K3) - weight of the criteria - 2 (L3).

The score for each EO is calculated using formulas listed bellow:

K=K1+K2+K3

K - the overlall score for qualification K1 - the score for the first qualification criteria K2 - the score for the second qualification criteria K3 - the score for the fird qualification criteria

K(i) = T(i) / T(i)max * L(i)

i – index of the qualification criteria; K(i) – the score for the qualification criteria of EO under calculation T(i) – value of the criteria of the of EO under evaluation; T(i)max – the best value of the criteria of EO from all EO under evaluation; L(i) – proportion rate (weight of the criteria).

The same calculation method is used for calculation of economicaly advantegeus tender.

[paulakeen]

Dear Zilvaras,

Thank you for your contribution. This is really interesting for the evaluation. This is why it was decided to not include the score in the current ESPD-EDM.

[paulakeen]

A new CodeList would be required to codify the types of evaluation. For the time being only two values are considered:

1. Pass/Fail
2. Weighted

[ec-mcs]

Marc-Christopher SCHMIDT: Hi, we will start at 13:04

Timo Rantane / Hansel / Finland: Hello from Helsinki

wojcida: Hi Marc, we are with Virgiliu

Steve Patterson (Scottish Government): Yes

Timo Rantane / Hansel / Finland: could you please mute your microphones until you have something to say. This would help all to hear better

Steve Patterson (Scottish Government): There are some fileds that are just information, so probably yes/

Steve Patterson (Scottish Government): Agree with Jose Luis for the selection criteria, the bidders should uderstand how it is to be evaluted

Zilvaras: Yes

Timo Rantane / Hansel / Finland: why would you have a criterion that is not used as pass/fail or used for scoring?

Timo Rantane / Hansel / Finland: otherwise the question is useless (one shouldn't ask questions that are not used as selection criterion)

Jose Luis Cueva - Spain: CAs might choose not imposing any criteria at all. Then the ESPD do not need to specify neither pass/fail nor weighed but then no Part IV is needed at all!

Jose Luis Cueva - Spain: Fully agree to Timo

Steve Patterson (Scottish Government): I also agree with Timo and Jose Luis

Marc-Christopher SCHMIDT: Timo will provide screenshots. Enric will work on the model and provide examples as well

Enric Staromiejski (GROW): Enric to provide also a new gc CodeList named EvaluationMethodCodes

[Zilvaras]

In case of restricted procedure Contracting Authorities in Lithuania can chose what financial / finantial ratio criterias to use for weighting of qualicication of EO. Decision must be based on oublic procurement principals, etc., but it is not directly forbiden.

[Zilvaras]

Note for Enric: It is used only for chossing 5 best candidates to be invited to tender. Best 5 candidates are selected according the score of qualification.

[ec-mcs]

Rudolf: IFor open procedures only the pass/fail criteria should be show.

Jose Luis Cueva: I think same as Rudolf, but I cannot rule out the opttion

Rudolf: In an open procedure the CA authority has to consider all tenders. The contract is awarded based on award criteria.

Giampaolo SELLITTO: weights are for the technical part

Giampaolo SELLITTO: except Not for admission criteria

Giampaolo SELLITTO: except in a two phase restricted procedure

Rudolf: For example there is on pass/fail criteria "Registration with an professional/trade register. Its not possible to weigh this criteria. Its a fact or not.

Giampaolo SELLITTO: I agree

Jose Luis Cueva: I agree

Cécile GUASCH: checking in the vademecuum weighting is indeed limited to award criteria

Rudolf: As soon the CA uses selection criteria which go beyond pass/fail the procedere is a two step procedure, no matter how it is named by the CA. Therefore criteria with weighting should not be available in an ESPD form used for an open procedure.

Rudolf: Pass/Failcriteria must be allowed for all procedures

Cécile GUASCH: SHould n't you differentaite selection criteria with additional criteria

Jose Luis Cueva: ESPD is only for Selection (pre-awarding)

Jose Luis Cueva: Selection of the "best offer" is beyond our scope

Rudolf: Awarding criteria should not be part of the ESPD. Otherwise it would be ncessary to introduce a new section titelsd "Awared criteria"

Giampaolo SELLITTO: away it is useful to have an extensible model that can support both admission and selection criteria

Giampaolo SELLITTO: The ESPD is for admission criteria

Cécile GUASCH: OK but shouldn't this be reflected in the requirements and the information about the selection criteria

Giampaolo SELLITTO: The Model could be reused, though

Rudolf: Pass/Fail criteria must be used for all procedures

Jose Luis Cueva: Requ 1: In an open procedure the CA has to consider all tenders THAT DO NOT FAIL THE SELECTION CRITERIA!

Rudolf: Jose Luis is right.

Rudolf: The Directive mentions only “Selection Criteria”. Infact there are two kinds of Selection Criteria and therefore also used different terms e.g. in Austrian procurement law:A) Suitability/Eligibility Criteria (Eignungskriterien) to prove the suitability of the Economic Operator (pass or fail criteria)versusB) Selection Criteria (Auswahlkriterien) to choose the best Economic Operators out of those who met the Suitability Criteria (e.g. the best five Economic Operators) – for a twostep procedure

Jose Luis Cueva: FIRST REQU SHOULD BE COMPLETED WITH THE CAPITALIZED PHRASE

Jose Luis Cueva: THE NEXT PHRASE

Giampaolo SELLITTO: ESPD is a standard form regulation by the regulation 2015 9

Jose Luis Cueva: THAT DO NOT FAIL THE SELECTION CRITERIA

Giampaolo SELLITTO: The directives has ESPD but there is also a part on the selection criteria

Jose Luis Cueva: ALL TENDERS ...THAT DO NOT FAIL THE SELECTION CRITERIA!

Giampaolo SELLITTO: not only admission, but no standard form for the submission of that part of the offer

Jose Luis Cueva: i THINK IT IS CORRECT

Jose Luis Cueva: I think so too

Rudolf: Please read my comment above.

Cécile GUASCH: Reduction criteria are also used in the Commission

Cécile GUASCH: They may be called Additional selection criteria

[paulakeen]

According to the latest discussions, possible mock-ups for the weighting could be the ones below:

1) For the Contracting Authority:

2) For the Economic Operator:

[paulakeen]

According to the latest discussions the EDM UML Model would have the aspect of the figure below. Noticed that 2 new elements have been added: WeightScoringMethodology and WeightingType.

The Criterion Element has also been modified to include three new elements, EvaluationType, EvaluationInformation and CriterionWeighting.

The XSD Schemata corresponding to these UML follows below, too:

1) Root element:

2) Criterion

[paulakeen]

The XML below are examples of how to use the new XSD Schemata according to the decisions made up to now (for version 1.1.0, ZOOM IN TO INCREASE READABILITY):

A simplified way of the structure implemented in this XML instance could be represented as follows:

1) Root element:

2) Criterion

[ec-mcs]

Deal done: Agreed in eMeeting

[paulakeen]

Please find the details of the final decision in https://github.com/ESPD/ESPD-EDM/wiki/4.-Weighting