Collecting additional structured information on bids and assessment process (Discussion)

[timgdavies]

A features extension is under consideration for the 1.1 version of OCDS.

This issue has been updated to reflect the current state of proposal development.

The issue

There are scenarios where:

• A tender process may request additional structured information from bidders;
• The procuring entity, or committee assessing bids, may need to provide structured judgements on each of the bids;
• Based on these, formulae may be applied to help calculate assessments of the bid, supporting judgements about the 'Most Economically Advantageous Proposal'

In a distributed system, where bidders are submitting data through different portals, an approach is required that can:

• Describe the structured data that should be provided and how it should be stored;
• Describe how this data should be attached to the OCDS release;
• Describe the formulae for calculating computed values;

The current approach being explored is providing a schema for this extra information which systems can use to provide additional structured data.

Specific requirements

Any approach should be able to:

• Describe who must provide the additional data (e.g. bidder, assessment committee etc.);
• Describe how the data should be stored;
• Describe whether the data should be published openly or not;

  Proposal

See https://github.com/open-contracting/standard/issues/223#issuecomment-129741289 for a draft proposal which involves:

• Creating a features array in tender, bids, award and contracts
• Allowing features to cross-reference to the supplier or the line-items of a contracting process

Further work to develop a fully worked up features extension will take place in coming weeks.

Engagement

Do you support the development of a features extension? Do the proposals in this thread head in the right direction?

[timgdavies]

@myroslav: When we discussed this you spoke of potentially storing the additional data as attached documents.

Is there a reason for not storing it within the JSON structure of a release itself?

Then the schemas could be simple JSON-Patch documents (much like an OCDS extension), and the calculation would be able to operate over a single JSON tree. Each JSON-Patch document for structured data could specify which user role was responsible for providing that data, and could potentially introduce a field for determining whether the contents of any given field should be published openly.

It might be useful as well if we can construct a very simple example case. For e.g. can you give an example of one or two extra fields a bidder would be asked to supply, and a few that the procuring agency would use, and then a calculation over them? That would let us start working up some example JSON for different approaches.

[myroslav]

Schema have to be stored in Tender itself. If it is document, then it is in Tender' document contents. If it is not in document contents, then it should be

• some string field with JSON schema or JSON-Patch object serialized in JSON
• JSON schema or JSON-Patch as subobject

After that every bid should have

• either fields according to JSON schema
  • in bid documentation
  • or in subobject
• or fields of subobject of Bid according to JSON-Patch.

The JSON-Patch has following drawbacks:

• the structure of bid is controllable (in security/evil sense) by tender creator
• it provides ability to alter other fields or even delete some
• its area of application should be limited to special custom fields subobject
• or the root of application should be that custom fields subobject

All these items above does not make JSON-Patch an elegant solution.

"JSON schema" approach solves all of the issues above, but still has following issues to resolve:

• is schema stored in string field, as structure or as document of Tender?
• are custom fields stored in subobject or in a document of bid?

[myroslav]

The most significant example fields in case of vacuum cleaner purchase are:

• electricity power level
• power of air intake
• warranty period
• class (consumer / industrial)

I.e. I do have minimal level of "air intake" power and minimal "guarantee period", but I can buy more expensive vacuum cleaner if it is more powerful, has longer warranty period or is more energy efficient.

[timgdavies]

Ok. Let's construct this as a worked example, and explore the options.

Here's my attempt at a worked example. Have I missed any key steps etc?

Step 1: A buyer issues a tender for a vacuum cleaner

In the tender release they specify the extra information that bids should contain, in addition to supplier information and price.

In this worked example we will ask for air intake and guarantee period.

Step 2: The bidders submit their bids including this additional information

The current bid submission approach used in ProZorro is here.

Step 3: The system validates this additional information and stores it

It should be able to reject incorrectly formed additional information.

Step 4: The system makes calculations based on this additional information (optional)

For example, calculating a score based on air intake, guarantee period and price.

Step 5: The system is able to re-publish this additional information

At such point as bids are made open.

[timgdavies]

Looking at this, I wonder if an alternative approach to the idea of pushing a JSON Schema out at Step 1 for the additional information, and then having potentially arbitrary data structures to store in the system at future stages, would be to abstract out all this additional information as features of an item.

I could envisage a features array attached to items that would looks something like this:

"items":[
  {
    "id": "0001",
    "description": "string",
    "classification": {...}
    "quantity": "08",
    "unit": {...}
    "features":[
        {
            "code":"OCDS-123454-AIR",
            "title":"Air Intake",
            "description":"The air intake of the vacuum cleaner",
            "type":"integer",
            "pattern":"[0-9]*",
        },
        {
            "code":"OCDS-123454-GUARANTEE",
            "title":"Guarantee Period (months)",
            "description":"The guarantee period for the vacuum cleaner in months.",
            "type":"integer",
            "pattern":"[0-9]*",
        },
        {
          "code":"OCDS-123454-GUARANTEE_TERMS",
          "title":"Guarantee Terms",
          "description":"An explanation of any specific guarantee terms that may apply.",
          "type":"string",
          "pattern":"",
        },
    ]
  }
]

Here I've use the JSON schema type property to indicate what type of field should be presented to the bidder at Step 1/2, and have mocked up use of the pattern property for validating inputs against a regex (some other validation options might be wanted).

Then, when submitting their data, a bid object could look something like this:

{
    "data": {
        "tenderers": [...],
        "value": {
            "amount": 480000
        },
        "features":[
            {
                "code":"OCDS-123454-AIR",
                "value":100
            },
            {
                "code":"OCDS-123454-GUARANTEE",
                "value":24
            },
            {
                "code":"OCDS-123454-GUARANTEE_TERMS",
                "value":"Products must be returned to the depot for assessment and repair."
            }
        ]
    }
}

Reflections

This approach is perhaps less flexible than arbitrary JSON schema - but could have a number of advantages:

• Easier to implement
• Feature codes could be made more easily re-usable (which could be very good for analysis across different processes, and discovering, for e.g., what the average length of guarantee offered by bidders on different kinds of products is.)
• Only requires a fairly small extension to OCDS

Limitations

In this suggestion I've not tried working out how calculation might work - and there are lots more details to look at in terms of how values etc. should be modelled, but would welcome feedback on this alternative approach.

[myroslav]

"Features" is elegant idea. But I'm afraid it but doesn't cover "Features" of provider itself. Not sure why but one can ask for something like "years on a market" since features of the provider itself is used to mean a lot when evaluating proposals. Not sure if it is good or bad, but it is as it is. This can be embedded in item, but that not the most elegant workaround.

[timgdavies]

So - two options here before we discount features.

(1) Attach features to the tender/bid level (rather than item)

But introduce new elements to them for something like featureOf and relatedItem or relatedTenderer

This would make for something like:

 "features":[
        {
            "code":"OCDS-123454-AIR",
            "featureOf":"item",
            "relatedItem":"0001"
            "title":"Air Intake",
            "description":"The air intake of the vacuum cleaner",
            "type":"integer",
            "pattern":"[0-9]*",
        },
        {
            "code":"OCDS-123454-YEARS",
            "featureOf":"tenderer"
            "title":"Years trading",
            "description":"The number of years the tendering organisation has been trading",
            "type":"integer",
            "pattern":"[0-9]*",
        }
]

the tender release, and then:

{
    "data": {
        "tenderers": [...],
        "value": {
            "amount": 480000
        },
        "features":[
            {
                "code":"OCDS-123454-AIR",
                "value":100,
                "releatedItem":"0001"
            },
            {
                "code":"OCDS-123454-YEARS",
                "value":5,
                "relatedTenderer":"13306232"
            }
       ]
    }
}

in the actual bid object.

(2) Allow features to be attached in a number of places - to both tenderers and items

Although I'm not sure where we would attach the features for a tenderer in the Step 1 tender release.

It might be that we need a general features block, but that that is related:

(a) To items via itemFeatures

(b) To tender via tendererFeatures

[timgdavies]

Aside: The other element that features could include would be a threshold or minimumCriteria - for example, to specific that a bidder must have been formed for at least N years before bidding.

[myroslav]

The case (1) with featureOf property and tenderer/item value with relatedItem if necessary is better in terms of scope, that "Features" proposal influence and allow further extensions, if necessary.

The threshold/minimumCrieria is interesting approach to itemize the requirements. However minimum is not proper measure since sometimes minimum is important sometimes maximum and sometimes there is acceptable interval. Thus some better validation approach then simple regex or simple minimumCriteria is necessary.

[myroslav]

What about source for the data about the Feature? I.e. difference between Bidder and Assessment Committee? Should we add source property with tenderer and procuringEntity as possible values?

[myroslav]

Should we establish separate field for name or add something like id for every Feature to be able to reference them in Formula?

[timgdavies]

I was thinking the code could be the way of referencing the feature - although I note that (a) in the example above the code is quite a long string; and (b) in the case where you have a single feature you want declared for two different items this would fall down; and (c) having id would help the merge strategies in OCDS work.

So - I think id makes a lot of sense.

That would make it something like:

 "features":[
        {
            "id":"air",
            "source":"tenderer",
            "code":"OCDS-123454-AIR",
            "featureOf":"item",
            "relatedItem":"0001"
            "title":"Air Intake",
            "description":"The air intake of the vacuum cleaner",
            "type":"integer",
            "pattern":"[0-9]*",
        },
    {
            "id":"air-2",
            "code":"OCDS-123454-AIR",
            "source":"tenderer",
            "featureOf":"item",
            "relatedItem":"0002"
            "title":"Air Intake",
            "description":"The air intake of the vacuum cleaner",
            "type":"integer",
            "pattern":"[0-9]*",
        },
        {
            "id":"trading",   
            "source":"tenderer",
            "code":"OCDS-123454-YEARS",
            "featureOf":"tenderer"
            "title":"Years trading",
            "description":"The number of years the tendering organisation has been trading",
            "type":"integer",
            "pattern":"[0-9]*",
        }
]

and for submissions:

{
    "data": {
        "tenderers": [...],
        "value": {
            "amount": 480000
        },
        "features":[
            {
                "id":"air"
                "value":100,
                "releatedItem":"0001"
            },
            {
                "id":"air-2"
                "value":100,
                "releatedItem":"0002"
            },
            {
                "id":"trading",
                "value":5,
                "relatedTenderer":"13306232"
            }
       ]
    }
}

[myroslav]

One (and only) of the types of values we'll have is the "choices" one. I.e. Feature is expected to list all possible numerical choices with their textual explanation (tile, and possibly also a description).

JSONschema does have 'enum' and 'oneOf' for the goal. Can you suggest something similar for Feature?

[timgdavies]

So, from looking at enum, in JSON schema the values in the enum array can be of any type.

So, we could simple add enum as a value of the features array, such that:

If type = 'string', and enum is an array of strings, then value of that feature should be one of those strings.

If type = 'array' and enum is an array of strings, then value of that feature should be an array of one or more of those strings.

If type = 'object' and enum is an array of objects, then valueof that feature should be a single object.

If type = 'array' and enum is an array of objects, then the value of that feature should be an array of one or more objects.

The objects in enum could then contain an id, title and description, and you could choose in your application logic whether you will enforced redundancy of the title and description when giving feature values, or whether you will just accept the ID.

How would that work? It avoids us needing to introduce oneOf, and I think is consistent with JSON Schema's use of the terms.

[myroslav]

In our case, the primary and only type is float, then we'd need something like:

      "enum": [
          {
              "title": "До 1000 Вт", 
              "value": 0.1
          }, 
          {
              "title": "Більше 1000 Вт", 
              "value": 0.15
          }
      ], 

I'm not sure that ID is necessary, since that can lead to some manipulations in following event sequence:

1. Tender is announced
2. Bids are placed (and IDs are used instead of values)
3. Tender is corrected and values in enum are changed (with IDs remaining the same)
4. Automagically values submitted become different

This would mean that "copying" value from "enum" into Bid is proper way from legal standpoint.

[timgdavies]

Makes sense: so we can leave the redundancy as recommended, but not specify in depth what should go within the enum lists - allowing different systems to use arbitrary objects.

Shall I work up an extension on this basis?

[timgdavies]

On the issue of assessment processes, just spotted http://compranetfacil.com/plataforma-nacional-de-transparencia-segunda-etapa which presents structured information on the bid process. May be relevant to this thread, or to related extensions.

[timgdavies]

From research into the kinds of information contractors find useful at debrief state:

Number of bidders Rankings for each What was valuable in the proposals

[timgdavies]

Notes from modelling an example from the GDS Digital Outcomes and Specialists Framework using a mock up of Features.

• Features might belong to the 'project' or 'process' overall
• There may be 'informational' features. I.e. where it is useful to describe features that a supplier or bid should have, but where the bid process does not collect this in a structured way (e.g. information might be picked out from narrative text of the bid)
• Some features might be weighted, others might not (could be useful to flag which are 'assessment' features)

[JachymHercher]

This thread seems to be bringing together quite a few things. Find some comments below, to see whether I've understood it correctly. By the way, wouldn't all of this be better as an "open codelist"? (One for uses 1 & 2 below, another one for use 3.)

1) Taking the first part of @myroslav's vacuum cleaner example:

I.e. I do have minimal level of "air intake" power and minimal "guarantee period"

Here we try to structure the description of the object of the contract. If I understand it correctly, that is the main goal of the "features" extension - adding it to tenders and bids, including minimal and maximal values.

2) Taking also the second half of the example:

but I can buy more expensive vacuum cleaner if it is more powerful, has longer warranty period or is more energy efficient.

Here we try to link the structure to quantifiable quality award criteria. Same as above, these award criteria should have a minimal/maximal acceptable value for the tender to be evaluated. (This type of things is very important for eAuctions, where competition can be on anything which is quantifiable.)

3) Contrary to award criteria, which are about the contract matter, things like "years on the market" are about the bidder, i.e. in this case we try to structure eligibility/selection criteria. (Years on the market, turnover, largest previous project, for example). Please note that quite a lot of things are happening in the EU on this side right now (classification of selection criteria, and in the future perhaps also full structuring) in the context of the European Single Procurement Document - see e.g. https://github.com/ESPD.

[timgdavies]

@JachymHercher Sorry for the slow reply - but I can confirm from your notes above that you have understood correctly what we're aiming to do with the features extension - covering a number of use cases with a flexible data model.

[timgdavies]

To check my understanding in return:

ESPD is the 'European Single Procurement Document' and represents essentially a form that suppliers would have to fill in to describe themselves, and whether they meet the criteria that a government entity has set out for the economics operators who might want to bid on an opportunity.

We could consider in developing the features extension how ESPD data would map into features.

[JachymHercher]

@timgdavies Concerning the ESPD yes, that's it.

Concerning the "features" extension, you are not approaching it as a codelist, right? You want to just describe the features, not classify them?

[timgdavies]

Correct. An implementation of the features extension could use codes from an external codelist to describe the kind of features covered, but we are not proposing any codelists at this point in time.

[timgdavies]

I've just been looking at the EU's Core Criterion and Core Evidence Vocabulary (CCCEV) which has recently issued Draft 4 for consultation.

The CCCEV has a model based on specifying requirements (like our 'features' concept) using Criterion which are fulfilled by one or more RequirementGroup which contain CriterionRequirements.

The use of RequirementGroup increases the expressiveness of the model, as a Criterion can be satisfied by any of it's attached requirement groups (OR), but the CriterionRequirements within a group must all be satisfied for the group to be satisfied (AND).

Responses to a criterion requirement are given by a RequirementResponse which can be linked to a PeriodOfTime during which it is valid, and to Evidence that supports the criterion.

There are also some interesting modelling steps in relation to the way in which CriterionRequirements (ranges etc.) can be specified, and CCCEV work may support development of re-usable criterion identifiers in future.

@duncandewhurst this would definitely be worth exploring in your work on features extension.

[duncandewhurst]

I've expanded slightly on the vacuum cleaner example given above and used the CCCEV approach and language of criteria, requirementGroups and requirements to illustrate the AND/OR functionality of CCEV.

This gives the following structure for specifying requirements at the tender stage:

"tender": {
    "criteria": [
        {
            "id": "0001",
            "title": "Air intake",
            "description": "The vacuum cleaner air intake must be at least 100W",
            "source": "tenderer",
            "code": "OCDS-123454-AIR",
            "featureOf": "item",
            "relatedItem": "0001",
            "requirementGroups": [
                {
                    "id": "0001-001",
                    "description": "The vacuum cleaner air intake must be at least 100W",
                    "requirements": [
                        {
                            "id": "0001-001-01",
                            "title": "Air intake",
                            "description": "Power of vacuum cleaner air intake in W",
                            "dataType": "integer",
                            "pattern": "[0-9]*",
                            "minValue": 100
                        }
                    ]
                }
            ]
        },
        {
            "id": "0002",
            "title": "Warranty",
            "description": "The vacuum cleaner must have warranty support options for at least 36 months",
            "source": "tenderer",
            "code": "OCDS-123454-WARRANTY",
            "featureOf": "item",
            "relatedItem": "0001",
            "requirementGroups": [
                {
                    "id": "0002-001",
                    "description": "The standard warranty period for the vacuum cleaner must be at least 36 months",
                    "requirements": [
                        {
                            "id": "0002-001-01",
                            "title": "Standard warranty period",
                            "description": "Length of the vacuum cleaner standard warranty period in months",
                            "dataType": "integer",
                            "pattern": "[0-9]*",
                            "minValue": 36
                        }
                    ]
                },
                {
                    "id": "0002-002",
                    "description": "The standard warranty period for the vacuum cleaner must be at least 12 months with an option to extend to 36 months",
                    "requirements": [
                        {
                            "id": "0002-002-01",
                            "title": "Standard warranty period",
                            "description": "Length of the vacuum cleaner standard warranty period in months",
                            "dataType": "integer",
                            "pattern": "[0-9]*",
                            "minValue": 12
                        },
                        {
                            "id": "0002-002-02",
                            "title": "Extended warranty option",
                            "description": "There is an extended warranty option for at least 36 months",
                            "dataType": "boolean",
                            "expectedValue": true
                        }
                    ]
                }
            ]
        },
        {
            "id": "0003",
            "title": "Years trading",
            "description": "Number of years the bidder has been trading",
            "source": "tenderer",
            "code": "OCDS-123454-YEARS",
            "featureOf": "tenderer",
            "requirementGroups": [
                {
                    "id": "0003-001",
                    "description": "Number of years the bidder has been trading",
                    "requirements": [
                        {
                            "id": "0003-001-01",
                            "title": "Years trading",
                            "description": "Number of years the bidder has been trading",
                            "dataType": "integer",
                            "pattern": "[0-9]*",
                            "minValue": 3
                        }
                    ]
                }
            ]
        }
    ]
}

And for responses:

"bids": [
    {
        "requirementResponses": [
            {
                "id": "air",
                "value": 125,
                "requirement": "0001-001-01",
                "relatedItem": "0001"
            },
            {
                "id": "warranty",
                "value": 36,
                "requirement": "0002-001-01",
                "relatedItem": "0001"
            },
            {
                "id": "years",
                "value": 10,
                "requirement": "0003-001-01"
            }
        ]
    }
]

In this approach the requirement.id would need to be unique within the OCID to allow cross referencing from the bids in which case we may be able to remove the relatedItem property of requirementResponse.

@myroslav - does this meet your requirements?

There are a couple of areas in the CCCEV model which differ from the OCDS approach:

• fulfilledIndicator at criteria level - according to the CCCEV documentation this flag is used by bidders to indicate whether they consider the criteria to be met. In the proposed approach above bidders would only provide requirementReponse level data so would not use this flag
• CCCEV has its own classes for certain concepts which OCDS already has a building block for, e.g. organizations, documents etc.

CCCEV has some additional detail which we currently expect to sit outside of the proposed features/requirements extension, although existing OCDS building blocks could be reused for most of these concepts:

• criteria.formalFramework - Used to reference the legal instrument which mandates the criteria
• evidence - Used to support requirementResponses and to specify where evidence is neccessary for certain requirements
• Additional properties of requirements including translationType and certificationLevel

Feedback and thoughts on the above very welcome.

We're also unsure of the wider applicability of the enum approach discussed in previous comments so would appreciate views on this from the community. We're considering whether this could be expressed as semi-structured data using the requirements.description field.

[myroslav]

The CCCEV-inspired proposal of @duncandewhurst is too verbose for ProZorro and MEAT (as outlined via Ukrainian Public Procurement Law).

Example of Features that are in ProZorro can be observed at http://api-docs.openprocurement.org/en/latest/tutorial.html#creating-tender (search for features). Tutorial has no example of the bid, but Bid data structures can be reviewed at http://api-docs.openprocurement.org/en/latest/standard/bid.html (see Bid.parameters and Parameter). Complementary Feature is described at http://api-docs.openprocurement.org/en/latest/standard/feature.html.

Formula that is weighting the price and parameters of every feature is hard-coded in the Public Procurement Law and perfectly aligns with several aspects of public procurement:

• procuring entities should be limited in flexibility (i.e. maximum weight of non-price criteria is 30%)
• bidder should be limited in choices (this reduces manual errors, can be compensated with more options in enum)
• weighting approach is same and clearly understood for all players (formula in the law)

More about weighting at http://openprocurement.org/en/nonprice-criteria.html

[timgdavies]

Thanks @myroslav: is the issue solely verbosity?

I.e. is it possible to model the information from your features in terms of a single criteria and requirementsGroup?

Or are there semantic issues we need to consider here?

[timgdavies]

The version of this extension for review is now located at https://github.com/open-contracting/ocds_requirements_extension

[ec-mcs]

We are currently updating the ESPD data model (https://github.com/ESPD/ESPD-EDM) which is behind the CCEV. The aim is to have the ESPD data model updated end of April. After that the idea is to CCEV accordingly. When do you want to have your new version ready?

[timgdavies]

@ec-mcs We're aiming to release OCDS 1.1 by end of April, which I think would be formally pinned to a version of our ocds_requirements_extension as it stands then, and also ocds_requirements_extension will be integrated into the Public Private Partnerships profile of OCDS which is due for release around the same time.

However, the extension can continue to evolve further beyond that for early adopters.

Where would we look to get a sense of any substantial changes anticipated to EPSD / CCEV data structures?

[JachymHercher]

My comment from the 1.1 peer review:

As discussed in #385, I take it that the requirements extension, same as CCCEV and ESPD, aims to model only exclusion grounds and selection criteria, not award criteria. If this is the case, this should be explicitly mentioned in the documentation.