ckan / ckanext-validation

CKAN extension for validating Data Packages using Table Schema.
MIT License
28 stars 32 forks source link

TypeError when editing ressources #90

Open amadzar opened 10 months ago

amadzar commented 10 months ago

I am Using Ckan 2.10.1 and just installed ckanext-validation. I have linked the required presets and the required presets scheming.dataset_schemas = ckanext.validation.examples:ckan_default_schema.json scheming.presets = ckanext.scheming:presets.json Ckanext.validation:presets.json Now when I click on an existing resource and even when I am not changing anything but clicking on "Update resource" I get this error: TypeError: str cannot be used as validator because it is not a user-defined function. Why is that and how can I fix this? I am quite new to the ckan world and it's also the first time working for me with ckanext-validation so maybe I am just missing something I don't know. Any help is much appreciated, thanks in advance!

ThrawnCA commented 10 months ago

What is your ckanext-scheming version? Older versions may cause this error.

Gutelvam commented 10 months ago

I'm encountering the same error even after a clean installation. I've attempted this with CKAN version 2.10.1 and Python versions 3.10, 3.8.13, and 3.9.18, and in all cases, I encountered the same error when attempting to upload a file. Unfortunately, I don't have any additional information to provide about the error at this time. Here's the content of my ckan.ini file:

#
# CKAN configuration
#
# These are some of the configuration options available for your CKAN
# instance. Check the documentation in 'doc/configuration.rst' or at the
# following URL for a description of what they do and the full list of
# available options:
#
# http://docs.ckan.org/en/latest/maintaining/configuration.html
#
# The %(here)s variable will be replaced with the parent directory of this file
#
[DEFAULT]

## Default settings ############################################################
## This enables the `Flask-DebugToolbar
## <https://flask-debugtoolbar.readthedocs.io/>`_ in the web interface, makes
## Webassets serve unminified JS and CSS files, and enables CKAN templates'
## debugging features.
## 
## You will need to ensure the ``Flask-DebugToolbar`` python package is installed,
## by activating your ckan virtual environment and then running::
## 
##     pip install -r /usr/lib/ckan/default/src/ckan/dev-requirements.txt
## 
## If you are running CKAN on Apache, you must change the WSGI
## configuration to run a single process of CKAN. Otherwise
## the execution will fail with: ``AssertionError: The EvalException
## middleware is not usable in a multi-process environment``. Eg. change::
## 
##   WSGIDaemonProcess ckan_default display-name=ckan_default processes=2 threads=15
##   to
##   WSGIDaemonProcess ckan_default display-name=ckan_default threads=15
## 
## .. warning:: This option should be set to ``False`` for a public site.
##    With debug mode enabled, a visitor to your site could execute malicious
##    commands.
debug = false

[app:main]

## General settings ############################################################
use = egg:ckan
ckan.legacy_route_mappings = {}
config.mode = default

## Development settings ########################################################
ckan.devserver.host = localhost
ckan.devserver.port = 5000
ckan.devserver.threaded = false
ckan.devserver.multiprocess = 1
ckan.devserver.watch_patterns = 
ckan.devserver.ssl_cert = 
ckan.devserver.ssl_key = 

## Session settings ############################################################
ckan.user.last_active_interval = 600
cache_dir = /tmp/%(ckan.site_id)s
beaker.session.key = ckan
beaker.session.secret = OcRYZS4yk-N2rknfMu7-CdipiI8
beaker.session.auto = false
beaker.session.cookie_expires = false
# beaker.session.cookie_domain = .example.com
beaker.session.save_accessed_time = true
beaker.session.secure = false
beaker.session.timeout = 600

## Database settings ###########################################################
sqlalchemy.url = postgresql://ckan_default:12345@localhost/ckan_default
sqlalchemy.pool_pre_ping = true
# sqlalchemy.<OPTION> = 

## Site Settings ###############################################################
ckan.site_url = http://127.0.0.1:5000
apikey_header_name = X-CKAN-API-Key
ckan.cache_expires = 0
ckan.cache_enabled = false
ckan.mimetype_guess = file_ext
ckan.static_max_age = 3600
ckan.tracking_enabled = false
ckan.valid_url_schemes = http https ftp
ckan.requests.timeout = 5
ckan.hide_version = false
ckan.redirect_to_login_if_not_authorized = true

## Authorization Settings ######################################################
ckan.auth.anon_create_dataset = false
ckan.auth.create_unowned_dataset = false
ckan.auth.create_dataset_if_not_in_organization = true
ckan.auth.user_create_groups = true
ckan.auth.user_create_organizations = true
ckan.auth.user_delete_groups = true
ckan.auth.user_delete_organizations = true
ckan.auth.create_user_via_api = false
ckan.auth.create_user_via_web = true
ckan.auth.roles_that_cascade_to_sub_groups = admin
ckan.auth.public_user_details = true
ckan.auth.public_activity_stream_detail = false
ckan.auth.allow_dataset_collaborators = false
ckan.auth.allow_admin_collaborators = false
ckan.auth.allow_collaborators_to_change_owner_org = false
ckan.auth.create_default_api_keys = false
ckan.auth.login_view = user.login
ckan.auth.reveal_private_datasets = false
ckan.auth.enable_cookie_auth_in_api = true
ckan.auth.route_after_login = dashboard.datasets

## CSRF Protection #############################################################
WTF_CSRF_ENABLED = true
WTF_CSRF_CHECK_DEFAULT = true
WTF_CSRF_SECRET_KEY = AM3LDx_t1OKaABxkFbh_4mEPKxrAeRg3Ou_LxquSBX0
WTF_CSRF_METHODS = POST PUT PATCH DELETE
WTF_CSRF_FIELD_NAME = _csrf_token
WTF_CSRF_HEADERS = X-CSRFToken X-CSRF-Token
WTF_CSRF_TIME_LIMIT = 3600
WTF_CSRF_SSL_STRICT = true
WTF_I18N_ENABLED = true
ckan.csrf_protection.ignore_extensions = true

## Flask-Login Remember me cookie settings #####################################
REMEMBER_COOKIE_NAME = remember_token
REMEMBER_COOKIE_DURATION = 31536000
# REMEMBER_COOKIE_DOMAIN = .example.com
REMEMBER_COOKIE_PATH = /
REMEMBER_COOKIE_SECURE = false
REMEMBER_COOKIE_HTTPONLY = true
REMEMBER_COOKIE_REFRESH_EACH_REQUEST = false
REMEMBER_COOKIE_SAMESITE = None

## API Token Settings ##########################################################
api_token.nbytes = 32
api_token.jwt.encode.secret = string:%(beaker.session.secret)s
api_token.jwt.decode.secret = string:%(beaker.session.secret)s
api_token.jwt.algorithm = HS256

## Search Settings #############################################################
ckan.site_id = default
solr_url = http://127.0.0.1:8983/solr/ckan
solr_user = 
solr_password = 
ckan.search.remove_deleted_packages = true
ckan.search.solr_commit = true
ckan.search.show_all_types = dataset
ckan.search.default_include_private = true
ckan.search.default_package_sort = score desc, metadata_modified desc
search.facets = organization groups tags res_format license_id
search.facets.limit = 50
search.facets.default = 10
ckan.extra_resource_fields = 
ckan.search.rows_max = 1000
ckan.group_and_organization_list_max = 1000
ckan.group_and_organization_list_all_fields_max = 25
solr_timeout = 60

## Redis Settings ##############################################################
ckan.redis.url = redis://localhost:6379/0

## CORS Settings ###############################################################
ckan.cors.origin_allow_all = false
ckan.cors.origin_whitelist = 

################ ______DATASTORE SETTINGS_____#############
ckan.datastore.write_url = postgresql://ckan_default:12345@localhost/datastore_default
ckan.datastore.read_url = postgresql://datastore_default:12345@localhost/datastore_default

###############______XLOADER SETTINGS______##############
ckanext.xloader.api_token = eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI4TFdWQjF6cFFnOGxvMmdxU19mU2czUGZfbF85ZUMwSWRGX2JBRjlaREE0IiwiaWF0IjoxNjk2OTcxNTYxfQ.F-myzVPpq0TMrZe-Scd1HbElSbt_D6OKoCdha1LWCqc
ckanext.xloader.jobs_db.uri = postgresql://ckan_default:12345@localhost/ckan_default

#############_______VALIDATION SETTINGS______##############
scheming.dataset_schemas = ckanext.validation.examples:ckan_default_schema.json
scheming.presets = ckanext.scheming:presets.json
                   ckanext.validation:presets.json
ckanext.validation.run_on_create_async = True
ckanext.validation.run_on_update_async = True
ckanext.validation.formats = csv xlsx
ckanext.validation.show_badges_in_listings = True
ckanext.validation.default_validation_options={
    "skip_errors": ["blank-row", "duplicate-label"],
    }

## Plugins Settings ############################################################
ckan.plugins = activity datastore xloader validation scheming_datasets
ckan.resource_proxy.timeout = 5

## Front-End Settings ##########################################################
ckan.site_title = CKAN
ckan.site_description = 
ckan.site_intro_text = 
ckan.site_logo = /base/images/ckan-logo.png
ckan.site_about = 
ckan.theme = css/main
ckan.favicon = /base/images/ckan.ico
ckan.datasets_per_page = 20
package_hide_extras = 
ckan.dumps_url = 
ckan.dumps_format = 
ckan.recaptcha.publickey = 
ckan.recaptcha.privatekey = 
ckan.featured_groups = 
ckan.featured_orgs = 
ckan.default_group_sort = title
ckan.gravatar_default = identicon
ckan.debug_supress_header = false
ckan.homepage_style = 1
ckan.site_custom_css = 

## Resource Views Settings #####################################################
ckan.views.default_views = image_view datatables_view

## Theming Settings ############################################################
ckan.template_head_end = 
ckan.template_footer_end = 
ckan.template_title_delimiter = -
extra_template_paths = 
extra_public_paths = 
ckan.base_public_folder = public
ckan.base_templates_folder = templates
ckan.default.package_type = dataset
ckan.default.group_type = group
ckan.default.organization_type = organization
ckan.admin_tabs = {}

## Storage Settings ############################################################
ckan.storage_path = /var/lib/ckan/default
ckan.max_resource_size = 10
ckan.max_image_size = 2

## Uploader Settings ###########################################################
ckan.upload.user.types = 
ckan.upload.user.mimetypes = 
ckan.upload.group.types = 
ckan.upload.group.mimetypes = 

## Webassets Settings ##########################################################
ckan.webassets.path = 
ckan.webassets.use_x_sendfile = false

## User Settings ###############################################################
ckan.user_list_limit = 20
ckan.user_reset_landing_page = home.index

## Activity Streams Settings ###################################################
ckan.activity_streams_enabled = true
ckan.activity_streams_email_notifications = false
ckan.activity_list_limit = 31
ckan.activity_list_limit_max = 100
ckan.email_notifications_since = 2 days
ckan.hide_activity_from_users = %(ckan.site_id)s

## Feeds Settings ##############################################################
ckan.feeds.author_name = 
ckan.feeds.author_link = 
ckan.feeds.authority_name = 
ckan.feeds.date = 
ckan.feeds.limit = 20

## Internationalisation Settings ###############################################
ckan.locale_default = en
ckan.locales_offered = 
ckan.locales_filtered_out = 
ckan.locale_order = 
ckan.i18n_directory = 
ckan.i18n.extra_directory = 
ckan.i18n.extra_gettext_domain = 
ckan.i18n.extra_locales = 
ckan.i18n.rtl_languages = he ar fa_IR
ckan.i18n.rtl_theme = css/main-rtl
ckan.display_timezone = UTC
ckan.root_path = 
ckan.resource_formats = /usr/lib/ckan/default/src/ckan/ckan/config/resource_formats.json

## Form Settings ###############################################################
ckan.dataset.create_on_ui_requires_resources = true
package_new_return_url = 
package_edit_return_url = 
licenses_group_url = 

## Email settings ##############################################################
smtp.server = localhost
smtp.starttls = false
smtp.user = 
smtp.password = 
smtp.mail_from = 
smtp.reply_to = 
email_to = 
error_email_from = 

## Background Job Settings #####################################################
ckan.jobs.timeout = 180

## Logging configuration
[loggers]
keys = root, ckan, ckanext, werkzeug

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = WARNING
handlers = console

[logger_werkzeug]
level = WARNING
handlers = console
qualname = werkzeug
propagate = 0

[logger_ckan]
level = INFO
handlers = console
qualname = ckan
propagate = 0

[logger_ckanext]
level = DEBUG
handlers = console
qualname = ckanext
propagate = 0

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s

I've double-checked the version of ckanext-scheming, and it appears to be up to date at version 3.0.0. Here is a list of the libraries installed in the environment:

alembic==1.8.1
attrs==23.1.0
Babel==2.10.3
Beaker==1.11.0
bleach==5.0.1
blinker==1.5
boto3==1.28.62
botocore==1.31.62
cached-property==1.5.2
certifi==2021.10.8
chardet==5.2.0
charset-normalizer==2.0.12
-e git+https://github.com/ckan/ckan.git@9915ba0022b9a74a65e61c097b2fee584b044087#egg=ckan
ckanapi==4.7
-e git+https://github.com/ckan/ckanext-scheming.git@8646a9dce79aa0b5a46274271ca6330dc0870b92#egg=ckanext_scheming
-e git+https://github.com/frictionlessdata/ckanext-validation.git@1073c80dace453a404df3d5f1ac1ae86a88b5029#egg=ckanext_validation
ckanext-xloader==0.12.2
ckantoolkit==0.0.7
click==8.1.3
colorama==0.4.6
Deprecated==1.2.13
docopt==0.6.2
dominate==2.7.0
et-xmlfile==1.1.0
feedgen==0.9.0
Flask==2.0.3
Flask-Babel==1.0.0
Flask-Login==0.6.1
flask-multistatic==1.0
Flask-WTF==1.0.1
frictionless==5.0.0b9
greenlet==1.1.2
humanize==4.8.0
idna==3.3
ijson==3.2.3
importlib-metadata==4.11.3
isodate==0.6.1
itsdangerous==2.1.1
Jinja2==3.1.2
jmespath==1.0.1
jsonlines==4.0.0
jsonschema==4.19.1
jsonschema-specifications==2023.7.1
linear-tsv==1.1.0
lxml==4.9.1
Mako==1.2.2
Markdown==3.4.1
markdown-it-py==3.0.0
marko==2.0.0
MarkupSafe==2.0.1
mdurl==0.1.2
mypy==0.971
mypy-extensions==0.4.3
nose==1.3.7
openpyxl==3.1.2
packaging==21.3
passlib==1.7.4
petl==1.7.14
polib==1.1.1
psycopg2==2.9.3
Pygments==2.16.1
PyJWT==2.4.0
pyparsing==3.0.7
pysolr==3.9.0
python-dateutil==2.8.2
python-magic==0.4.27
python-slugify==8.0.1
pytz==2021.3
pytz-deprecation-shim==0.1.0.post0
PyUtilib==6.0.0
PyYAML==6.0
redis==4.1.4
referencing==0.30.2
requests==2.31.0
rfc3986==2.0.0
rich==13.6.0
rpds-py==0.10.4
rq==1.11.0
s3transfer==0.7.0
shellingham==1.5.3
simpleeval==0.9.13
simplejson==3.17.6
six==1.16.0
SQLAlchemy==1.4.41
sqlalchemy2-stubs==0.0.2a27
sqlparse==0.4.2
stringcase==1.2.0
tableschema==1.20.2
tabulate==0.9.0
tabulator==1.53.5
text-unidecode==1.3
tomli==2.0.1
typer==0.9.0
typing_extensions==4.3.0
tzdata==2022.1
tzlocal==4.2
unicodecsv==0.14.1
Unidecode==1.0.22
urllib3==1.26.9
validators==0.22.0
watchdog==2.1.6
webassets==2.0
webencodings==0.5.1
Werkzeug==2.0.3
wrapt==1.14.0
WTForms==3.0.1
xlrd==2.0.1
zipp==3.7.0
zope.interface==5.4.0

and for information, here is the error mensage when i try to upload something:

**2023-10-10 17:11:54,461 INFO  [ckan.config.middleware.flask_app]  304 /base/vendor/select2/select2.png render time 0.005 seconds
2023-10-10 17:12:03,877 ERROR [ckan.config.middleware.flask_app] str cannot be used as validator because it is not a user-defined function
Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 246, in convert
    nargs = converter._code_.co_argcount
AttributeError: type object 'str' has no attribute '_code_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python3.10/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/ckan/default/lib/python3.10/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/usr/lib/ckan/default/lib/python3.10/site-packages/flask/views.py", line 84, in view
    return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)
  File "/usr/lib/ckan/default/lib/python3.10/site-packages/flask/views.py", line 158, in dispatch_request
    return current_app.ensure_sync(meth)(*args, **kwargs)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware/../../views/resource.py", line 377, in post
    get_action(u'resource_update')(context, data)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/_init_.py", line 551, in wrapped
    result = _action(context, data_dict, **kw)
  File "/usr/lib/ckan/default/src/ckanext-validation/ckanext/validation/logic.py", line 544, in resource_update
    return up_func(context, data_dict)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/update.py", line 111, in resource_update
    updated_pkg_dict = _get_action('package_update')(context, pkg_dict)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/_init_.py", line 551, in wrapped
    result = _action(context, data_dict, **kw)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/update.py", line 311, in package_update
    data, errors = lib_plugins.plugin_validate(
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/plugins.py", line 327, in plugin_validate
    result = plugin.validate(context, data_dict, schema, action)
  File "/usr/lib/ckan/default/src/ckanext-scheming/ckanext/scheming/plugins.py", line 312, in validate
    return navl_validate(data_dict, schema, context)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 305, in validate
    flat_data, errors = _validate(flattened, schema, validators_context)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 356, in _validate
    convert(converter, key, converted_data, errors, context)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 248, in convert
    raise TypeError(
TypeError: str cannot be used as validator because it is not a user-defined function**

Given this situation, I would greatly appreciate any assistance in resolving this issue. Thank you very much.

ThrawnCA commented 10 months ago

The example schema references the unicode validator when it should be unicode_safe.