search

ckan / ckanext-validation

CKAN extension for validating Data Packages using Table Schema.
MIT License
28 stars 33 forks source link

TypeError: str cannot be used as validator because it is not a user-defined function #91

Open Gutelvam opened 1 year ago

Gutelvam commented 1 year ago

I'm encountering this error even after a clean installation. I've attempted this with CKAN version 2.10.1, 2.9.9 and Python versions 3.10, 3.8.13, and 3.9.18, and in all cases, I encountered the same error when attempting to upload a file. Unfortunately, I don't have any additional information to provide about the error at this time. Here's the content of my ckan.ini file:

#
# CKAN configuration
#
# These are some of the configuration options available for your CKAN
# instance. Check the documentation in 'doc/configuration.rst' or at the
# following URL for a description of what they do and the full list of
# available options:
#
# http://docs.ckan.org/en/latest/maintaining/configuration.html
#
# The %(here)s variable will be replaced with the parent directory of this file
#
[DEFAULT]

## Default settings ############################################################
## This enables the `Flask-DebugToolbar
## <https://flask-debugtoolbar.readthedocs.io/>`_ in the web interface, makes
## Webassets serve unminified JS and CSS files, and enables CKAN templates'
## debugging features.
## 
## You will need to ensure the ``Flask-DebugToolbar`` python package is installed,
## by activating your ckan virtual environment and then running::
## 
##     pip install -r /usr/lib/ckan/default/src/ckan/dev-requirements.txt
## 
## If you are running CKAN on Apache, you must change the WSGI
## configuration to run a single process of CKAN. Otherwise
## the execution will fail with: ``AssertionError: The EvalException
## middleware is not usable in a multi-process environment``. Eg. change::
## 
##   WSGIDaemonProcess ckan_default display-name=ckan_default processes=2 threads=15
##   to
##   WSGIDaemonProcess ckan_default display-name=ckan_default threads=15
## 
## .. warning:: This option should be set to ``False`` for a public site.
##    With debug mode enabled, a visitor to your site could execute malicious
##    commands.
debug = false

[app:main]

## General settings ############################################################
use = egg:ckan
ckan.legacy_route_mappings = {}
config.mode = default

## Development settings ########################################################
ckan.devserver.host = localhost
ckan.devserver.port = 5000
ckan.devserver.threaded = false
ckan.devserver.multiprocess = 1
ckan.devserver.watch_patterns = 
ckan.devserver.ssl_cert = 
ckan.devserver.ssl_key = 

## Session settings ############################################################
ckan.user.last_active_interval = 600
cache_dir = /tmp/%(ckan.site_id)s
beaker.session.key = ckan
beaker.session.secret = OcRYZS4yk-N2rknfMu7-CdipiI8
beaker.session.auto = false
beaker.session.cookie_expires = false
# beaker.session.cookie_domain = .example.com
beaker.session.save_accessed_time = true
beaker.session.secure = false
beaker.session.timeout = 600

## Database settings ###########################################################
sqlalchemy.url = postgresql://ckan_default:12345@localhost/ckan_default
sqlalchemy.pool_pre_ping = true
# sqlalchemy.<OPTION> = 

## Site Settings ###############################################################
ckan.site_url = http://127.0.0.1:5000
apikey_header_name = X-CKAN-API-Key
ckan.cache_expires = 0
ckan.cache_enabled = false
ckan.mimetype_guess = file_ext
ckan.static_max_age = 3600
ckan.tracking_enabled = false
ckan.valid_url_schemes = http https ftp
ckan.requests.timeout = 5
ckan.hide_version = false
ckan.redirect_to_login_if_not_authorized = true

## Authorization Settings ######################################################
ckan.auth.anon_create_dataset = false
ckan.auth.create_unowned_dataset = false
ckan.auth.create_dataset_if_not_in_organization = true
ckan.auth.user_create_groups = true
ckan.auth.user_create_organizations = true
ckan.auth.user_delete_groups = true
ckan.auth.user_delete_organizations = true
ckan.auth.create_user_via_api = false
ckan.auth.create_user_via_web = true
ckan.auth.roles_that_cascade_to_sub_groups = admin
ckan.auth.public_user_details = true
ckan.auth.public_activity_stream_detail = false
ckan.auth.allow_dataset_collaborators = false
ckan.auth.allow_admin_collaborators = false
ckan.auth.allow_collaborators_to_change_owner_org = false
ckan.auth.create_default_api_keys = false
ckan.auth.login_view = user.login
ckan.auth.reveal_private_datasets = false
ckan.auth.enable_cookie_auth_in_api = true
ckan.auth.route_after_login = dashboard.datasets

## CSRF Protection #############################################################
WTF_CSRF_ENABLED = true
WTF_CSRF_CHECK_DEFAULT = true
WTF_CSRF_SECRET_KEY = AM3LDx_t1OKaABxkFbh_4mEPKxrAeRg3Ou_LxquSBX0
WTF_CSRF_METHODS = POST PUT PATCH DELETE
WTF_CSRF_FIELD_NAME = _csrf_token
WTF_CSRF_HEADERS = X-CSRFToken X-CSRF-Token
WTF_CSRF_TIME_LIMIT = 3600
WTF_CSRF_SSL_STRICT = true
WTF_I18N_ENABLED = true
ckan.csrf_protection.ignore_extensions = true

## Flask-Login Remember me cookie settings #####################################
REMEMBER_COOKIE_NAME = remember_token
REMEMBER_COOKIE_DURATION = 31536000
# REMEMBER_COOKIE_DOMAIN = .example.com
REMEMBER_COOKIE_PATH = /
REMEMBER_COOKIE_SECURE = false
REMEMBER_COOKIE_HTTPONLY = true
REMEMBER_COOKIE_REFRESH_EACH_REQUEST = false
REMEMBER_COOKIE_SAMESITE = None

## API Token Settings ##########################################################
api_token.nbytes = 32
api_token.jwt.encode.secret = string:%(beaker.session.secret)s
api_token.jwt.decode.secret = string:%(beaker.session.secret)s
api_token.jwt.algorithm = HS256

## Search Settings #############################################################
ckan.site_id = default
solr_url = http://127.0.0.1:8983/solr/ckan
solr_user = 
solr_password = 
ckan.search.remove_deleted_packages = true
ckan.search.solr_commit = true
ckan.search.show_all_types = dataset
ckan.search.default_include_private = true
ckan.search.default_package_sort = score desc, metadata_modified desc
search.facets = organization groups tags res_format license_id
search.facets.limit = 50
search.facets.default = 10
ckan.extra_resource_fields = 
ckan.search.rows_max = 1000
ckan.group_and_organization_list_max = 1000
ckan.group_and_organization_list_all_fields_max = 25
solr_timeout = 60

## Redis Settings ##############################################################
ckan.redis.url = redis://localhost:6379/0

## CORS Settings ###############################################################
ckan.cors.origin_allow_all = false
ckan.cors.origin_whitelist = 

################ ______DATASTORE SETTINGS_____#############
ckan.datastore.write_url = postgresql://ckan_default:12345@localhost/datastore_default
ckan.datastore.read_url = postgresql://datastore_default:12345@localhost/datastore_default

###############______XLOADER SETTINGS______##############
ckanext.xloader.api_token = eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI4TFdWQjF6cFFnOGxvMmdxU19mU2czUGZfbF85ZUMwSWRGX2JBRjlaREE0IiwiaWF0IjoxNjk2OTcxNTYxfQ.F-myzVPpq0TMrZe-Scd1HbElSbt_D6OKoCdha1LWCqc
ckanext.xloader.jobs_db.uri = postgresql://ckan_default:12345@localhost/ckan_default

#############_______VALIDATION SETTINGS______##############
scheming.dataset_schemas = ckanext.validation.examples:ckan_default_schema.json
scheming.presets = ckanext.scheming:presets.json
                   ckanext.validation:presets.json
ckanext.validation.run_on_create_async = True
ckanext.validation.run_on_update_async = True
ckanext.validation.formats = csv xlsx
ckanext.validation.show_badges_in_listings = True
ckanext.validation.default_validation_options={
    "skip_errors": ["blank-row", "duplicate-label"],
    }

## Plugins Settings ############################################################
ckan.plugins = activity datastore xloader validation scheming_datasets
ckan.resource_proxy.timeout = 5

## Front-End Settings ##########################################################
ckan.site_title = CKAN
ckan.site_description = 
ckan.site_intro_text = 
ckan.site_logo = /base/images/ckan-logo.png
ckan.site_about = 
ckan.theme = css/main
ckan.favicon = /base/images/ckan.ico
ckan.datasets_per_page = 20
package_hide_extras = 
ckan.dumps_url = 
ckan.dumps_format = 
ckan.recaptcha.publickey = 
ckan.recaptcha.privatekey = 
ckan.featured_groups = 
ckan.featured_orgs = 
ckan.default_group_sort = title
ckan.gravatar_default = identicon
ckan.debug_supress_header = false
ckan.homepage_style = 1
ckan.site_custom_css = 

## Resource Views Settings #####################################################
ckan.views.default_views = image_view datatables_view

## Theming Settings ############################################################
ckan.template_head_end = 
ckan.template_footer_end = 
ckan.template_title_delimiter = -
extra_template_paths = 
extra_public_paths = 
ckan.base_public_folder = public
ckan.base_templates_folder = templates
ckan.default.package_type = dataset
ckan.default.group_type = group
ckan.default.organization_type = organization
ckan.admin_tabs = {}

## Storage Settings ############################################################
ckan.storage_path = /var/lib/ckan/default
ckan.max_resource_size = 10
ckan.max_image_size = 2

## Uploader Settings ###########################################################
ckan.upload.user.types = 
ckan.upload.user.mimetypes = 
ckan.upload.group.types = 
ckan.upload.group.mimetypes = 

## Webassets Settings ##########################################################
ckan.webassets.path = 
ckan.webassets.use_x_sendfile = false

## User Settings ###############################################################
ckan.user_list_limit = 20
ckan.user_reset_landing_page = home.index

## Activity Streams Settings ###################################################
ckan.activity_streams_enabled = true
ckan.activity_streams_email_notifications = false
ckan.activity_list_limit = 31
ckan.activity_list_limit_max = 100
ckan.email_notifications_since = 2 days
ckan.hide_activity_from_users = %(ckan.site_id)s

## Feeds Settings ##############################################################
ckan.feeds.author_name = 
ckan.feeds.author_link = 
ckan.feeds.authority_name = 
ckan.feeds.date = 
ckan.feeds.limit = 20

## Internationalisation Settings ###############################################
ckan.locale_default = en
ckan.locales_offered = 
ckan.locales_filtered_out = 
ckan.locale_order = 
ckan.i18n_directory = 
ckan.i18n.extra_directory = 
ckan.i18n.extra_gettext_domain = 
ckan.i18n.extra_locales = 
ckan.i18n.rtl_languages = he ar fa_IR
ckan.i18n.rtl_theme = css/main-rtl
ckan.display_timezone = UTC
ckan.root_path = 
ckan.resource_formats = /usr/lib/ckan/default/src/ckan/ckan/config/resource_formats.json

## Form Settings ###############################################################
ckan.dataset.create_on_ui_requires_resources = true
package_new_return_url = 
package_edit_return_url = 
licenses_group_url = 

## Email settings ##############################################################
smtp.server = localhost
smtp.starttls = false
smtp.user = 
smtp.password = 
smtp.mail_from = 
smtp.reply_to = 
email_to = 
error_email_from = 

## Background Job Settings #####################################################
ckan.jobs.timeout = 180

## Logging configuration
[loggers]
keys = root, ckan, ckanext, werkzeug

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = WARNING
handlers = console

[logger_werkzeug]
level = WARNING
handlers = console
qualname = werkzeug
propagate = 0

[logger_ckan]
level = INFO
handlers = console
qualname = ckan
propagate = 0

[logger_ckanext]
level = DEBUG
handlers = console
qualname = ckanext
propagate = 0

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s

I've double-checked the version of ckanext-scheming, and it appears to be up to date at version 3.0.0. Here is a list of the libraries installed in the environment:

alembic==1.8.1
attrs==23.1.0
Babel==2.10.3
Beaker==1.11.0
bleach==5.0.1
blinker==1.5
boto3==1.28.62
botocore==1.31.62
cached-property==1.5.2
certifi==2021.10.8
chardet==5.2.0
charset-normalizer==2.0.12
-e git+https://github.com/ckan/ckan.git@9915ba0022b9a74a65e61c097b2fee584b044087#egg=ckan
ckanapi==4.7
-e git+https://github.com/ckan/ckanext-scheming.git@8646a9dce79aa0b5a46274271ca6330dc0870b92#egg=ckanext_scheming
-e git+https://github.com/frictionlessdata/ckanext-validation.git@1073c80dace453a404df3d5f1ac1ae86a88b5029#egg=ckanext_validation
ckanext-xloader==0.12.2
ckantoolkit==0.0.7
click==8.1.3
colorama==0.4.6
Deprecated==1.2.13
docopt==0.6.2
dominate==2.7.0
et-xmlfile==1.1.0
feedgen==0.9.0
Flask==2.0.3
Flask-Babel==1.0.0
Flask-Login==0.6.1
flask-multistatic==1.0
Flask-WTF==1.0.1
frictionless==5.0.0b9
greenlet==1.1.2
humanize==4.8.0
idna==3.3
ijson==3.2.3
importlib-metadata==4.11.3
isodate==0.6.1
itsdangerous==2.1.1
Jinja2==3.1.2
jmespath==1.0.1
jsonlines==4.0.0
jsonschema==4.19.1
jsonschema-specifications==2023.7.1
linear-tsv==1.1.0
lxml==4.9.1
Mako==1.2.2
Markdown==3.4.1
markdown-it-py==3.0.0
marko==2.0.0
MarkupSafe==2.0.1
mdurl==0.1.2
mypy==0.971
mypy-extensions==0.4.3
nose==1.3.7
openpyxl==3.1.2
packaging==21.3
passlib==1.7.4
petl==1.7.14
polib==1.1.1
psycopg2==2.9.3
Pygments==2.16.1
PyJWT==2.4.0
pyparsing==3.0.7
pysolr==3.9.0
python-dateutil==2.8.2
python-magic==0.4.27
python-slugify==8.0.1
pytz==2021.3
pytz-deprecation-shim==0.1.0.post0
PyUtilib==6.0.0
PyYAML==6.0
redis==4.1.4
referencing==0.30.2
requests==2.31.0
rfc3986==2.0.0
rich==13.6.0
rpds-py==0.10.4
rq==1.11.0
s3transfer==0.7.0
shellingham==1.5.3
simpleeval==0.9.13
simplejson==3.17.6
six==1.16.0
SQLAlchemy==1.4.41
sqlalchemy2-stubs==0.0.2a27
sqlparse==0.4.2
stringcase==1.2.0
tableschema==1.20.2
tabulate==0.9.0
tabulator==1.53.5
text-unidecode==1.3
tomli==2.0.1
typer==0.9.0
typing_extensions==4.3.0
tzdata==2022.1
tzlocal==4.2
unicodecsv==0.14.1
Unidecode==1.0.22
urllib3==1.26.9
validators==0.22.0
watchdog==2.1.6
webassets==2.0
webencodings==0.5.1
Werkzeug==2.0.3
wrapt==1.14.0
WTForms==3.0.1
xlrd==2.0.1
zipp==3.7.0
zope.interface==5.4.0

and for information, here is the error mensage when i try to upload something:

**2023-10-10 17:11:54,461 INFO  [ckan.config.middleware.flask_app]  304 /base/vendor/select2/select2.png render time 0.005 seconds
2023-10-10 17:12:03,877 ERROR [ckan.config.middleware.flask_app] str cannot be used as validator because it is not a user-defined function
Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 246, in convert
    nargs = converter._code_.co_argcount
AttributeError: type object 'str' has no attribute '_code_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python3.10/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/ckan/default/lib/python3.10/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/usr/lib/ckan/default/lib/python3.10/site-packages/flask/views.py", line 84, in view
    return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)
  File "/usr/lib/ckan/default/lib/python3.10/site-packages/flask/views.py", line 158, in dispatch_request
    return current_app.ensure_sync(meth)(*args, **kwargs)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware/../../views/resource.py", line 377, in post
    get_action(u'resource_update')(context, data)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/_init_.py", line 551, in wrapped
    result = _action(context, data_dict, **kw)
  File "/usr/lib/ckan/default/src/ckanext-validation/ckanext/validation/logic.py", line 544, in resource_update
    return up_func(context, data_dict)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/update.py", line 111, in resource_update
    updated_pkg_dict = _get_action('package_update')(context, pkg_dict)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/_init_.py", line 551, in wrapped
    result = _action(context, data_dict, **kw)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/update.py", line 311, in package_update
    data, errors = lib_plugins.plugin_validate(
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/plugins.py", line 327, in plugin_validate
    result = plugin.validate(context, data_dict, schema, action)
  File "/usr/lib/ckan/default/src/ckanext-scheming/ckanext/scheming/plugins.py", line 312, in validate
    return navl_validate(data_dict, schema, context)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 305, in validate
    flat_data, errors = _validate(flattened, schema, validators_context)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 356, in _validate
    convert(converter, key, converted_data, errors, context)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 248, in convert
    raise TypeError(
TypeError: str cannot be used as validator because it is not a user-defined function**

Given this situation, I would greatly appreciate any assistance in resolving this issue. Thank you very much.

Originally posted by @Gutelvam in https://github.com/frictionlessdata/ckanext-validation/issues/90#issuecomment-1756296738

ThrawnCA commented 1 year ago

My first advice is to rotate the Beaker secret and CSRF secret on whichever environment this came from, since they're publicly exposed. Also the API token.

ThrawnCA commented 1 year ago

Second, the example schema references the 'unicode' validator. That's where the problem comes from.

Gutelvam commented 1 year ago

My first advice is to rotate the Beaker secret and CSRF secret on whichever environment this came from, since they're publicly exposed. Also the API token.

First of all, thanks for the advice. However, it doesn't impact anything since it's a local environment on a VirtualBox VM machine for testing purposes. Second, thanks for the answer. I'll try to fix it.