search

italia / ckan-it

This project provides everything you need to run CKAN plus a set of extensions for supporting Italian open data in a set of Docker images.
Other
17 stars 19 forks source link

Errore su ckanext.multilang.model.package_multilang durante harvesting #56

Open wlorenzetti opened 4 years ago

wlorenzetti commented 4 years ago

Slave sto testando harvesting su un motore di dati RNDT che abbiamo sviluppato, durante un harvesting di tipo DCAT_AP-IT CSW Harvester ottengo il seguente errore:

2020-03-26 07:34:24,921 INFO  [ckanext.dcatapit.harvesters.csw_harvester] Medatata harvested dataset languages: 'ITA'
/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:79: SAWarning: Usage of the 'related attribute set' operation is not currently supported within the execution stage of the flush process. Results may not be consistent.  Consider using alternative event listeners or connection-level operations instead.
  sess._flush_warning("related attribute set")
2020-03-26 07:34:25,055 DEBUG [ckanext.spatial.plugin] Received: '{"type": "Polygon", "coordinates": [[[-20.664783, 36.316963], [64.219734, 36.316963], [64.219734, 70.433136], [-20.664783, 70.433136], [-20.664783, 36.316963]]]}'
2020-03-26 07:34:25,088 DEBUG [ckanext.spatial.lib] Created new extent for package 3a00e6d6-1d10-4a0e-a38f-3745d166ad03
2020-03-26 07:34:25,091 DEBUG [ckanext.dcatapit.interfaces] Creating create_loc_field for package ID: '3a00e6d6-1d10-4a0e-a38f-3745d166ad03'
2020-03-26 07:34:25,096 ERROR [ckanext.multilang.model.package_multilang] Exception occurred while persisting DB objects: (IntegrityError) null value in column "text" violates not-null constraint
DETAIL:  Failing row contains (6, 3a00e6d6-1d10-4a0e-a38f-3745d166ad03, publisher_name, extra, en, null).
 'INSERT INTO package_multilang (package_id, field, field_type, lang, text) VALUES (%(package_id)s, %(field)s, %(field_type)s, %(lang)s, %(text)s) RETURNING package_multilang.id' {'lang': 'en', 'text': None, 'package_id': u'3a00e6d6-1d10-4a0e-a38f-3745d166ad03', 'field': u'publisher_name', 'field_type': 'extra'}

Traceback (most recent call last):
  File "/usr/bin/paster", line 10, in <module>
    sys.exit(run())
  File "/usr/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 193, in command
    fetch_callback(consumer, method, header, body)
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 407, in fetch_callback
    fetch_and_import_stages(harvester, obj)
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 424, in fetch_and_import_stages
    success_import = harvester.import_stage(obj)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/harvesters/base.py", line 609, in import_stage
    package_id = p.toolkit.get_action('package_create')(context, package_dict)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 431, in wrapped
    result = _action(context, data_dict, **kw)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/create.py", line 207, in package_create
    item.after_create(context, data)
  File "/usr/lib/ckan/default/src/ckanext-dcatapit/ckanext/dcatapit/plugin.py", line 305, in after_create
    self.create_loc_field(extra, lang, pkg_dict.get('id'))
  File "/usr/lib/ckan/default/src/ckanext-dcatapit/ckanext/dcatapit/plugin.py", line 468, in create_loc_field
    interfaces.save_extra_package_multilang({'id': pkg_id, 'text': extra.get('value'), 'field': extra.get('key')}, lang, 'extra')
  File "/usr/lib/ckan/default/src/ckanext-dcatapit/ckanext/dcatapit/interfaces.py", line 83, in save_extra_package_multilang
    PackageMultilang.persist(pkg, lang, field_type)
  File "/usr/lib/ckan/default/src/ckanext-multilang/ckanext/multilang/model/package_multilang.py", line 151, in persist
    session.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 149, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 768, in commit
    self.transaction.commit()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 370, in commit
    self._prepare_impl()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 339, in _prepare_impl
    self.session.dispatch.before_commit(self.session)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 257, in __call__
    fn(*args, **kw)
  File "/usr/lib/ckan/default/src/ckan/ckan/model/meta.py", line 73, in before_commit
    session.flush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1907, in flush
    self._flush(objects)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2025, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 57, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1989, in _flush
    flush_context.execute()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 371, in execute
    rec.execute(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 524, in execute
    uow
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
    mapper, table, insert)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 600, in _emit_insert_statements
    execute(statement, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 727, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 824, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 954, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1116, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 947, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 435, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) null value in column "text" violates not-null constraint
DETAIL:  Failing row contains (6, 3a00e6d6-1d10-4a0e-a38f-3745d166ad03, publisher_name, extra, en, null).
 'INSERT INTO package_multilang (package_id, field, field_type, lang, text) VALUES (%(package_id)s, %(field)s, %(field_type)s, %(lang)s, %(text)s) RETURNING package_multilang.id' {'lang': 'en', 'text': None, 'package_id': u'3a00e6d6-1d10-4a0e-a38f-3745d166ad03', 'field': u'publisher_name', 'field_type': 'extra'}
2020-03-26 07:34:31,827 INFO  [ckanext.multilang.harvesters.multilang] CSW Multilang harvester: extending ISODocument with PT_FreeText
2020-03-26 07:34:32,246 DEBUG [ckanext.spatial.plugin] Setting up the spatial model
2020-03-26 07:34:32,287 DEBUG [ckanext.spatial.model.package_extent] Spatial tables defined in memory
2020-03-26 07:34:32,294 DEBUG [ckanext.spatial.model.package_extent] Spatial tables already exist
2020-03-26 07:34:32,308 DEBUG [ckanext.harvest.model] Harvest tables defined in memory
2020-03-26 07:34:32,310 DEBUG [ckanext.harvest.model] Harvest tables already exist
2020-03-26 07:34:32,455 DEBUG [ckanext.harvest.queue] Fetch queue consumer registered

Sto usando il docker-compose.yml attuale. Dove protrebbe essere il problema? Grazie a buon lavoro.

LME1209 commented 4 years ago

Ciao, per caso hai risolto questo problema?