search

open-contracting / lib-cove-ocds

A data review library for the Open Contracting Data Standard (OCDS)
Other
0 stars 0 forks source link

CLI fails with duplicate items #104

Closed Ravf95 closed 1 year ago

Ravf95 commented 2 years ago

The CLI fails when input uses an extension that contains an array of items with duplicate identifiers (e.g bid extension)

libcoveocds log ``` Traceback (most recent call last): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/bin/libcoveocds", line 11, in load_entry_point('libcoveocds==0.11.0', 'console_scripts', 'libcoveocds')() File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/click-8.1.3-py3.8.egg/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/click-8.1.3-py3.8.egg/click/core.py", line 1055, in main rv = self.invoke(ctx) File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/click-8.1.3-py3.8.egg/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/click-8.1.3-py3.8.egg/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/libcoveocds-0.11.0-py3.8.egg/libcoveocds/cli/__main__.py", line 64, in process File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/libcoveocds-0.11.0-py3.8.egg/libcoveocds/api.py", line 99, in ocds_json_output File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/libcoveocds-0.11.0-py3.8.egg/libcoveocds/common_checks.py", line 123, in common_checks_ocds File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/libcove-0.27.0-py3.8.egg/libcove/lib/common.py", line 595, in common_checks_context validation_errors = get_schema_validation_errors( File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/libcove-0.27.0-py3.8.egg/libcove/lib/common.py", line 808, in get_schema_validation_errors for e in our_validator.iter_errors(json_data): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/_validators.py", line 282, in properties for error in validator.descend( File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 344, in descend for error in self.iter_errors(instance, schema): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/_legacy_validators.py", line 55, in items_draft3_draft4 for error in validator.descend(item, items, path=index): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 344, in descend for error in self.iter_errors(instance, schema): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/_validators.py", line 263, in ref for error in validator.descend(instance, resolved): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 344, in descend for error in self.iter_errors(instance, schema): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/_validators.py", line 282, in properties for error in validator.descend( File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 344, in descend for error in self.iter_errors(instance, schema): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/_validators.py", line 263, in ref for error in validator.descend(instance, resolved): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 344, in descend for error in self.iter_errors(instance, schema): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/_validators.py", line 282, in properties for error in validator.descend( File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 344, in descend for error in self.iter_errors(instance, schema): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/_legacy_validators.py", line 55, in items_draft3_draft4 for error in validator.descend(item, items, path=index): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 344, in descend for error in self.iter_errors(instance, schema): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/_validators.py", line 263, in ref for error in validator.descend(instance, resolved): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 344, in descend for error in self.iter_errors(instance, schema): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/_validators.py", line 282, in properties for error in validator.descend( File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 344, in descend for error in self.iter_errors(instance, schema): File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/jsonschema-3.2.0-py3.8.egg/jsonschema/validators.py", line 328, in iter_errors for error in errors: File "../py/lib-cove-ocds/0__11__0/lib-cove-ocds-0.11.0/.ve/lib/python3.8/site-packages/libcove-0.27.0-py3.8.egg/libcove/lib/common.py", line 137, in unique_ids err = ValidationError(msg, instance=", ".join(non_unique_id)) TypeError: sequence item 0: expected str instance, int found ```
example ``` { "uri": "https://www.example.com", "publisher": { "name": "Example" }, "extensions": [ "https://raw.githubusercontent.com/open-contracting-extensions/ocds_bid_extension/master/extension.json" ], "publishedDate": "2010-05-10T09:30:00Z", "license": "http://opendatacommons.org/licenses/pddl/1.0/", "publicationPolicy": "https://github.com/open-contracting/sample-data/", "version": "1.1", "releases": [ { "ocid": "ocds-213czf-000-00001", "id": "ocds-213czf-000-00001-04-planning", "date": "2010-05-10T09:30:00Z", "language": "en", "tag": [ "planning" ], "initiationType": "tender", "bids": { "details": [ { "items": [ { "id": 1 }, { "id": 1 } ] } ] } } ] } ```

I'm not sure why this is happening, also I tested with other core fields that using item definition but the output is Non-unique id values (works fine)

cc @jpmckinney

Ravf95 commented 2 years ago

DRT fails with the example https://standard.open-contracting.org/review/data/b7bec8fb-eae2-40bb-9335-77328c1e5dd6 image

Ravf95 commented 2 years ago

Also, kingfisher process saves this error as sequence item 0: expected str instance, int found on release_check_error and record_check_error tables

jpmckinney commented 2 years ago

It's a bug in lib-cove. It expects the non-unique IDs to be str, but in this case they are int. It needs to coerce the values to str at https://github.com/OpenDataServices/lib-cove/blob/b86ac38c63e94a1262554a13e4c8ac9d95fa8a20/libcove/lib/common.py#L137

You'll need to make a PR against lib-cove first, then upgrade the version once the fix is released.