Default validation (ignore missing) is not uniformally applied

[Patrick1Rhode]

**The issue comes out when we try to create a package via the API.

Some missing validator over certain field types are causing SQL exception (please refer to the cases after the stack).**

Here is the example of an error based on a field called 'contact_email' which is defined without the ignore missing validator:

/usr/lib/ckan/default/bin/python /usr/lib/ckan/default/bin/paster serve --reload /etc/ckan/default/development.ini Starting subprocess with file monitor /usr/lib/ckan/default/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release. from cryptography import x509 2021-04-16 14:21:55,267 INFO [ckan.config.environment] Loading static files from public 2021-04-16 14:21:55,375 INFO [ckan.config.environment] Loading templates from /home/patrick/ckan/lib/default/src/ckan/ckan/templates 2021-04-16 14:21:57,143 INFO [ckan.config.environment] Loading templates from /home/patrick/ckan/lib/default/src/ckan/ckan/templates Starting server in PID 14947. serving on 0.0.0.0:5000 view at http://127.0.0.1:5000 2021-04-16 14:22:03,566 INFO [ckan.views.api] Validation error (Action API): "{'name': [u'That login name is not available.'], u'type': u'Validation Error'}" 2021-04-16 14:22:03,814 INFO [ckan.views.api] Validation error (Action API): "{'name': [u'That login name is not available.'], u'type': u'Validation Error'}" 2021-04-16 14:22:04,062 INFO [ckan.views.api] Validation error (Action API): "{'name': [u'That login name is not available.'], u'type': u'Validation Error'}" /usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py:2181: SAWarning: Usage of the 'related attribute set' operation is not currently supported within the execution stage of the flush process. Results may not be consistent. Consider using alternative event listeners or connection-level operations instead. % method) 2021-04-16 14:22:06,425 ERROR [ckan.views.api] (psycopg2.ProgrammingError) can't adapt type 'Missing' [SQL: 'INSERT INTO package_extra (id, package_id, key, value, state, revision_id) VALUES (%(id)s, %(package_id)s, %(key)s, %(value)s, %(state)s, %(revision_id)s)'] [parameters: {'package_id': 'dcd38414-7d22-4445-96d7-e9eb9cebaa10', 'value': <ckan.lib.navl.dictization_functions.Missing object at 0x7fbcd350f110>, 'state': 'active', 'key': 'contact_email', 'revision_id': u'99df0a2e-c1e7-490d-9771-b3dbd602334a', 'id': u'0b357b14-2989-4421-b712-6fdf51eb1e0e'}] Traceback (most recent call last): File "/home/patrick/ckan/lib/default/src/ckan/ckan/views/api.py", line 288, in action result = function(context, request_data) File "/home/patrick/ckan/lib/default/src/ckan/ckan/logic/init__.py", line 471, in wrapped result = _action(context, data_dict, *kw) File "/home/patrick/ckan/lib/default/src/fao-maps-ckanext-authorization/ckanext/fao_auth/logic/action.py", line 66, in package_create return next_auth(context, data_dict) File "/home/patrick/ckan/lib/default/src/ckan/ckan/logic/action/create.py", line 194, in package_create model.Session.flush() File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 157, in do return getattr(self.registry(), name)(args, **kwargs) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2171, in flush self._flush(objects) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2291, in _flush transaction.rollback(_capture_exception=True) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in exit compat.reraise(exc_type, exc_value, exc_tb) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2255, in _flush flush_context.execute() File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute rec.execute(self) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute uow File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj mapper, table, insert) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 835, in _emit_insert_statements execute(statement, params) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute return meth(self, multiparams, params) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement compiled_sql, distilled_params File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context context) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception exc_info File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context context) File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute cursor.execute(statement, parameters) ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'Missing' [SQL: 'INSERT INTO package_extra (id, package_id, key, value, state, revision_id) VALUES (%(id)s, %(package_id)s, %(key)s, %(value)s, %(state)s, %(revision_id)s)'] [parameters: {'package_id': 'dcd38414-7d22-4445-96d7-e9eb9cebaa10', 'value': <ckan.lib.navl.dictization_functions.Missing object at 0x7fbcd350f110>, 'state': 'active', 'key': 'contact_email', 'revision_id': u'99df0a2e-c1e7-490d-9771-b3dbd602334a', 'id': u'0b357b14-2989-4421-b712-6fdf51eb1e0e'}]

**1. If you have a multiple_text in YAML you will get an error if you don't add that field to the API and if you have not set validator to ignore_missing

2. For the multiple_checkbox, you don't need to set a validator. The API request will not raise an SQL exception even though you have not passed a multiple_checkbox field in the request
3. For select, you don't need to set a validator. The API request will not raise the SQL exception even though you have not passed a select field in the request
4. Email preset requires an ignore validator set; if not, the API request without it specified will raise SQL exception
5. URL preset also requires an ignore validator set; if not, the API request without it specified an SQL exception will be raised
6. And lastly, a regular field without any particular preset does not present any issues whether specified in the request the API request an SQL exception will not be raised
7. Even if we have one or more of the above field types without ignore validation (YAML definition) into the resource and we do not send the resource through the API the request will pass.
8. The case 7, will not pass if we send also the resource with the API request and not sending all the 'required' fields.**

What we want to highlight is that the behavior of normal fields, presets and complex fields are not always the same. Based on the normal behavior fields We were expecting that ignore missing is the default for all.

test_scripty and yaml.docx

[ccancellieri]

Actually now setting the validation to ignore missing I'm getting:

So now it's impossible to use any validation on those fields, I think we should give priority to: https://github.com/ckan/ckanext-scheming/issues/276

[ccancellieri]

related to https://github.com/ckan/ckanext-scheming/issues/286

[jmbouffard]

Is there a fix for this? This issue also impacts "multiple_text" items located on a 2nd page of the form using the "start_form_page:" feature. When switching to next page the CKAN server completely crashes with an error "can't adapt type 'Missing'".

A fix to the crash is to use the "validators: ignore_missing" on the "multiple_text" fields but then I get the wrong behavior that was reported in the previous messages (see example below).

[wardi]

Would you able to work on a fix, add a test that demonstrates the issue or provide a minimal schema that demonstrates the issue?

[jmbouffard]

Here is a minimal schema that will reproduce the issue as soon as you click on "next" button after filling out the first page of the dataset test_dataset_issue_286.zip

However I may have found a workaround. It seems that if I add the "validators: ignore_missing" to the preset directly instead of the data field in the yaml file, the error does not happen and the data is well entered in the database.

"preset_name": "multiple_text",
      "values": {
        "form_snippet": "multiple_text.html",
        "display_snippet": "multiple_text.html",
        "validators": "ignore_missing scheming_multiple_text",
        "output_validators": "scheming_load_json"

So is it possible that a validator specified in the yaml file would completely supersede the validator from the preset? If that is the case then it would explain why the workaround works.