KeyError: 'user' when harvesting ESRI DCAT document

[keithamoss]

I'm not completely sure if this is a ckanext-dcat issue or further upstream, so I'm starting at the bottom :)

This is running CKAN 2.2 on a 64bit AWS Linux AMI base and ckanext-harvest commands are being invoked by hand e.g.

paster --plugin=ckanext-harvest harvester fetch_consumer --config='/etc/ckan/default/production.ini'

et cetera

In using ckanext-dcat to harvest a DCAT document (http://vicroadsopendata.vicroadsmaps.opendata.arcgis.com/data.json) from ESRI (see: http://doc.arcgis.com/en/open-data/provider/federating-with-ckan.htm) ckanext-harvest's gather_consumer is retrieving 18 records successfully.

...
2015-05-14 06:06:31,794 DEBUG [ckanext.dcat.harvesters.base] Getting file http://vicroadsopendata.vicroadsmaps.opendata.arcgis.com/data.json?page=2
2015-05-14 06:06:32,787 DEBUG [ckanext.dcat.harvesters.base] Empty document, no more records
2015-05-14 06:06:32,790 DEBUG [ckanext.harvest.queue] Received from plugin gather_stage: 18 objects (first: [u'305d43cb-951d-4b1b-9bf8-40fa62d5f184'] last: [u'991585f9-ef83-4c1c-9922-4f19bc5c7c03'])
2015-05-14 06:06:32,793 DEBUG [ckanext.harvest.queue] Sent 18 objects to the fetch queue

But I'm running into this error from ckanext-harvest's fetch_consumer component.

Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 136, in command
    fetch_callback(consumer, method, header, body)
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 294, in fetch_callback
    fetch_and_import_stages(harvester, obj)
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 311, in fetch_and_import_stages
    success_import = harvester.import_stage(obj)
  File "/usr/lib/ckan/default/src/ckanext-dcat/ckanext/dcat/harvesters/base.py", line 347, in import_stage
    package_id = p.toolkit.get_action('package_create')(context, package_dict)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 420, in wrapped
    result = _action(context, data_dict, **kw)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/create.py", line 187, in package_create
    model.repo.commit()
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/vdm/sqlalchemy/tools.py", line 102, in commit
    self.session.commit()
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/scoping.py", line 114, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 656, in commit
    self.transaction.commit()
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 314, in commit
    self._prepare_impl()
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 290, in _prepare_impl
    self.session.dispatch.before_commit(self.session)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/event.py", line 291, in __call__
    fn(*args, **kw)
  File "/usr/lib/ckan/default/src/ckan/ckan/model/extension.py", line 112, in before_commit
    methodcaller('before_commit', session)
  File "/usr/lib/ckan/default/src/ckan/ckan/model/extension.py", line 92, in notify_observers
    func(observer)
  File "/usr/lib/ckan/default/src/ckan/ckan/model/modification.py", line 47, in before_commit
    self.notify(obj, domain_object.DomainObjectOperation.new)
  File "/usr/lib/ckan/default/src/ckan/ckan/model/modification.py", line 79, in notify
    observer.notify(entity, operation)
  File "/usr/lib/ckan/default/src/ckan/ckanext/datapusher/plugin.py", line 103, in notify
    'resource_id': entity.id
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 420, in wrapped
    result = _action(context, data_dict, **kw)
  File "/usr/lib/ckan/default/src/ckan/ckanext/datapusher/logic/action.py", line 51, in datapusher_submit
    user = p.toolkit.get_action('user_show')(context, {'id': context['user']})
KeyError: 'user'

I'm not nearly deep enough into how CKAN extensions knit together with the CKAN core to determine which extension should be supplying the user in this instance.

[amercader]

@keithamoss This was a bug on CKAN core (https://github.com/ckan/ckan/pull/1500) that was fixed and backported to the 2.2.x line. You should always run the latest patch release for CKAN (currently 2.2.2).

This should be straightforward, just follow this instructions to update:

http://docs.ckan.org/en/latest/maintaining/upgrading/upgrade-package-to-patch-release.html

Hope this helps!

[jnordling]

I was able to go through all the steps with the latest version of CKAN, attempting to import a dcat json format from ESRI Open Data portal. The problem I am having is that the import does not actually happen. The CKAN Harvest job tells me the "job" is running but nothing ever happens from there. Any advice on what were missing ?

http://doc.arcgis.com/en/open-data/provider/federating-with-ckan.htm

I get this error when running: paster --plugin=ckanext-harvest harvester fetch_consumer --config='/path/to/your config.ini'

Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 97, in command
    self._load_config()
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 156, in _load_config
    super(Harvester, self)._load_config()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 86, in _load_config
    load_environment(conf.global_conf, conf.local_conf)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 135, in load_environment
    p.load_all(config)
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 95, in load_all
    find_user_plugins(config)
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 177, in find_user_plugins
    raise PluginNotFoundException(name)
ckan.plugins.core.PluginNotFoundException: text_view