Codelist updates: awardCriteria

timgdavies commented 8 years ago

Updates to the awardCriteria codelist are being considered for version 1.1 of OCDS.

This builds upon discussions in #254.

The issue

The current open codelist for awardCriteria consists of:

lowestCost
bestProposal
bestValueToGovernment
singleBidOnly

This codelist lacks definitions.

The code bestValueToGovernment should not include ‘government’ within the code name, to support use of the standard in other contexts.

There is ambiguity between bestProposal and bestValueToGovernment.

When the award criteria is based on weighting of technical and price evaluation criteria, there is no way of representing this in a structured way.

The proposal

The codelist will be updated.

However, work remains to be done to derive a new codelist.

Engagement

We are inviting suggestions for what the awardCriteria codelist should contain.

What codes are required by key use cases? (e.g. Red Flag Analysis?)
Are there existing codelists we can defer to here?

myroslav commented 8 years ago

I'd asked for MEAT, where procuring entity is listing features, and their options and value associated with each option. Bidder then provides parameters of their bid within options offered. After that bids are evaluated on the MEAT criteria, where bid value is only part of total bid value, other part consist of parameters of features requested for buyer.

JachymHercher commented 8 years ago

I find the approach taken in http://simap.ted.europa.eu/documents/10184/99173/EN_F03.pdf (section II.2.5) to be quite good - submitting the individual criteria with a classification, not asking about the legal regimes (as they are not really relevant - the criteria are everything that matters and the difference between a MEAT with 99% on value and "lowest price" is confusing at best.)

Changes I would propose compared to the EU form is that "price" should be explicitly included in every procurement (in EU forms it is sometimes subsumed under cost) + perhaps include the options to use fixed prices and costs and/or unit prices and costs.

EU definitions: price refers to the acquisition price. Cost refers to any other costs that the contracting body wants to take into account, e.g. running costs, switching costs, disposal costs (i.e. other monetary values than the price). Quality criterion refers to any non-price non-cost criterion.

duncandewhurst commented 8 years ago

Feedback from the codelists breakout session at IODC:

Least cost was suggested by two attendees as an alternative to Lowest cost (Least cost is the term used in Nigeria)
The UNCITRAL model law on procurement was recommended as a possible source for definitions
Best value could be considered as a product of Lowest cost and Best proposal
Single bid was considered confusing
Most Economically Advantageous Tender (as defined in EU procurement directive 24) was suggested as an alternative to Best Proposal
Lowest cost criteria is no longer used in Scotland
Single criteria and Multiple criteria were also suggested as possible codes

duncandewhurst commented 8 years ago

Feedback from consultation with Hunt La Cascia, Senior Procurement Specialist at The World Bank:

"MEAT" or "most advantageous" (per the World Bank Procurement Regulations is typically used in place of "best value" or "best proposal"
"SingleBid" is ambiguous - does this meandirect selection (goods works and non-consulting services) or single selection (consulting services) or does it mean only one bid was received in a competitive process

duncandewhurst commented 8 years ago

Feedback from Alexandre Borges de Oliveira, World Bank:

Award criteria can be very granular but the key buckets would be:

lowestPrice
qualityOnly (common for consulting)
quality&Cost (most common for consulting)
ratedCriteria (equivalent to quality&Cost but typically used for goods and works and implies a weighting of multiple criteria)

The following terms are equivalent:

value for money
MEAT / most advantageous
best proposal
best value

JachymHercher commented 8 years ago

"qualityOnly (common for consulting)" - I would assume that this still means that there is a fixed price. (The information on which we propose to store, in the EU Forms, in the award criteria section.)

Concerning equivalent terms, in the new procurement directives, MEAT has been renamed to BPQR: "best price-quality ratio". On the contrary, "MEAT" is now just a theoretical concept which concerns every procurement procedure, with the options being "MEAT assessed on the basis of price or cost" or "MEAT assessed on the basis of price or cost and BPQR".

myroslav commented 7 years ago

Is ratedCriteria the value to use for awardCriteria in our procedures that include non-price criteria described at http://openprocurement.org/en/nonprice-criteria.html ?

timgdavies commented 7 years ago

We've not set the proposed updated list here - but it looks to me like ratedCriteria does fit the way your procedures are described.

timgdavies commented 7 years ago

Based on the discussions here I would propose the following codes and definitions:

Code	Title	Description	Deprecated
lowestPrice	Lowest Price	The award will be made to the qualified bid with the lowest price.
qualityOnly	Quality Only	The award will be made to the qualified bid demonstrating the highest quality against some assessment method.
qualityAndPrice	Quality and Price	The award will be made to the qualified bid demonstrating the highest score against some assessment method incorporating quality and price.
ratedCriteria	Rated Criteria	The award will be made to the qualified bid demonstrating the highest score against a set of weighted price, cost and quality criteria. Structured information on criteria can be provided with the requirements extension.
lowestCost	Lowest Cost	The award will be made to the qualified bid demonstrating the lowest overall cost. A cost assessment will cover the entire monetary implications of the proposal, including the price paid to the supplier and the running costs, switching costs or other non-price costs of choosing a particular option.
bestProposal	Best Proposal	This code has been deprecated. Please choose from one of the other codes.	1.1
bestValueToGovernment	Best Value to Government	This code has been deprecated. 'Rated criteria' or 'qualityAndCost' are alternatives.	1.1
singleBidOnly	Single Bid Only	Undefined. Code deprecated in 1.1.	1.1

This adds four new codes, and deprecates three.

LindseyAM commented 7 years ago

Based on review of the above and the expert comments provided by Myroslav, Alex, & Jachym, I suggest removing qualityAndPrice as it can be included within definition of ratedCriteria and possibility for extension where there are specific local formulas in use. I added a small twist for ‘other criteria’ such as in situations where there is a weighting criteria for eg some social or environmental dimension. Agree with the proposed deprecations.

Code	Title	Description	Deprecated
lowestPrice	Lowest Price	The award will be made to the qualified bid with the lowest price.
qualityOnly	Quality Only	The award will be made to the qualified bid demonstrating the highest quality against some assessment method.
~~qualityAndPrice~~	~~Quality and Price~~	~~The award will be made to the qualified bid demonstrating the highest score against some assessment method incorporating quality and price.~~
ratedCriteria	Rated Criteria	The award will be made to the qualified bid demonstrating the highest score against a set of weighted criteria such as price, cost and quality or other criteria eg best price-quality ratio or criteria that take social or environmental factors into account in the evaluation. Structured information on criteria can be provided with the requirements extension.
lowestCost	Lowest Cost	The award will be made to the qualified bid demonstrating the lowest overall cost. A cost assessment could cover the entire monetary implications of the proposal, including the price paid to the supplier and the running costs, switching costs or other non-price costs of choosing a particular option.
bestProposal	Best Proposal	This code has been deprecated. Please choose from one of the other codes.	1.1
bestValueToGovernment	Best Value to Government	This code has been deprecated. 'Rated criteria' or 'qualityAndCost' are alternatives.	1.1
singleBidOnly	Single Bid Only	Undefined. Code deprecated in 1.1.	1.1

timgdavies commented 7 years ago

Thanks @LindseyAM - I agree with this change. Will work on staging this shortly.

JachymHercher commented 7 years ago

Small naming comment: wouldn't renaming "qualityOnly" to "highestQuality" make it more consistent with "lowestPrice", "lowestCost"? (Or renaming the last two to priceOnly, costOnly.)

(By the way, as mentioned above, I still think that directly submitting information about criteria instead of "types of criteria" is better [you need just one layer / one codelist; not two]; I think the description of "quality only" needs to mention what happens with value, otherwise it's not clear how it should work.)

timgdavies commented 7 years ago

Thanks @JachymHercher - good points on code consistency - I've updated to have ratedCriteria, priceOnly, costOnly and qualityOnly

I agree that ideal is for people to provide information on each of the requirements (which I understand could be quality, price and cost) but at the moment that only exists in OCDS via an extension, and for analysis being able to read the summary off this codelist I think can still be useful.

So, the updated proposed codelist is now below:

Code	Title	Description	Deprecated
ratedCriteria	Rated Criteria	The award will be made to the qualified bid demonstrating the highest score against a set of weighted criteria such as price, cost and quality or other criteria eg best price-quality ratio or criteria that take social or environmental factors into account in the evaluation. Structured information on criteria can be provided with the requirements extension.
priceOnly	Price only	The award will be made to the qualified bid with the lowest price.
costOnly	costOnly	The award will be made to the qualified bid demonstrating the lowest overall cost. A cost assessment could cover the entire monetary implications of the proposal, including the price paid to the supplier and the running costs, switching costs or other non-price costs of choosing a particular option.
qualityOnly	Quality Only	The award will be made to the qualified bid demonstrating the highest quality against some assessment method. The price is either fixed, or with a maximum set and price factors not included in the evaluation.
bestProposal	Best Proposal	This code has been deprecated. Please choose from one of the other codes.	1.1
bestValueToGovernment	Best Value to Government	This code has been deprecated. 'Rated criteria' or 'qualityAndCost' are alternatives.	1.1
singleBidOnly	Single Bid Only	Undefined. Code deprecated in 1.1.	1.1

JachymHercher commented 7 years ago

@timgdavies, so the idea of the summary is having some information about the award criteria even if the users want to keep it simple / don't want to use an extension for describing them, right?

I agree with the idea of having a "simple" option, but the increase in complexity, if you ask directly about the criteria, is not that large. Essentially, instead of one selection from a codelist, users have to submit two weights, for price and cost, which gives you the same information as the whole codelist. (With cost being, imo, frequently 0%.) (The extension then adds also more detailed criteria and weights for individual quality criteria etc.)

Anyway, up to you:).

timgdavies commented 7 years ago

Ah - think I understand you now.

Ok - so idea is that instead of 'awardCriteria' as a single field, it could be:

"awardCriteria":{
     "price":0.4,
     "quality":0.6,
     "cost":0
}

which can carry the same information as the four codes above. Right?

I had been thinking the choice was between: (a) simple codelist; and (b) list all the criteria.

The issue with the approach above, in the context of the 1.1 upgrade, is that it would not be backwards compatible - which we aim for in integer upgrades.

So - I think my proposal would be to stick with the codelist update in 1.1, but to open an issue to do more work on this subsequently - particularly based on a review of how people to end up using the requirements extension.

JachymHercher commented 7 years ago

Yes. (You don't necessarily need all three though, just two - e.g. price & cost - are enough if you are willing to assume that the weighting adds up to 1 / 100%).

Nope, essentially it's a) just price & cost criteria b) all criteria, including price & cost.

Sure 👍 .

siwhitehouse commented 7 years ago

@JachymHercher, can you say whether concession contracts are relevant here, please?

From my reading of the TED notices and eForms Technical Specification then in the case of concession contracts there is the possibility of using an "order of importance" instead of a weighting and I wondered if that has any impact on the rated criteria proposed here.

JachymHercher commented 7 years ago

In eForms we are proposing to use the criteria in the same way for concessions as other types of contracts. Let's see if we come up with some reason why this could not work in the coming months.

Ordering is legally possible also for other types of public contracts. I'm doubtful that this is really used much in practice, so I wouldn't add it here for now - at least not until you have some clear demand from users.

JachymHercher commented 7 years ago

My comment from the 1.1 peer review:

1) The description for ratedCriteria states that "Structured information on criteria can be provided with the requirements extension." Reading the requirements extension I am not sure how I would use it for this purpose. Furthermore, the requirements extension, while somewhat complicated, does not have a basic yet crucial feature - a typology of award criteria, consisting of price-, cost-, and quality- criteria. Since the CCCEV, on which the requirements extension is based, explicitly covers only exclusion grounds and selection criteria, not award criteria, I think the requirements section is not really intended for award criteria.

This is very important information and fairly simple to model, so I would suggest adding soon a separate extension for modelling this. The approach proposed for the future EU forms could be used as inspiration.

2) I think the text "The award will be made to the qualified bid demonstrating the highest score against a set of weighted criteria such as price, cost and quality or other criteria eg best price-quality ratio or criteria that take social or environmental factors into account in the evaluation. Structured information on criteria can be provided with the requirements extension."

could be simplified to the text below. The point is not to mention BPQR (as it is simply another way of saying "ratedCriteria") and not separating environmental and social criteria from the price/cost/quality typology.

"The award will be made to the qualified bid demonstrating the highest score against a set of weighted criteria such as price, cost and quality (including, for example, the environmental and social impact of the procurement). Structured information on individual criteria can be provided with the requirements extension."

timgdavies commented 7 years ago

One comment from the peer review stated:

There are cases where "singleBidOnly" is an option for awarding the contract. Government of Canada Open Contracting data (buyandsell) currently uses this field, and if this field is deprecated we don't have a replacement for it.

I've reviewed to try and find a definition of 'singleBidOnly' as an award criteria (i.e. something set out in advance of a solicitation, rather than than post-hoc description of how awarding took place), and I've not managed to find one.

I can see it is possible in Canada for a contract to be awarded on the basis of 'One Responsible Bid' - but even here the award criteria would be on the basis of whether the price is competitive etc.

I think in this case we would need to review new mappings for the value currently being expressed using a singleBidOnly code.

JachymHercher commented 7 years ago

Out of curiosity - when would it be useful to have award criteria for a procurement process with only one bidder?

(To me it seems this use case is met by setting procurementMethod to direct and leaving the award criteria empty.)

open-contracting / standard