Algunos jobs están fallando

[defvol]

El harvester de SEDESOL y el de SCT se quedaron atorados en el fetch_queue y el log dice lo siguiente:

% (mapperutil.state_str(state), state.key)) sqlalchemy.exc.InvalidRequestError: Can't attach instance <Package at 0x579c8d0>; another instance with key (<class 'ckan.model.package.Package'>, (u'f4171abb-2dcc-4962-abdd-9d5ceba4c89d',)) is already present in this session.

[poguez]

Hay que esperar a que ocurra un error nuevamente para evaluar los logs.

[defvol]

[admin@app-datos-gob-mx ~]$ sudo supervisorctl [sudo] password for admin: ckan_fetch_consumer FATAL Exited too quickly (process log may have details) ckan_gather_consumer RUNNING pid 437, uptime 58 days, 23:08:59 supervisor> exit

[defvol]

Log

[admin@app-datos-gob-mx ~]$ tail -n 100 /var/log/ckan/default/fetch_consumer.log

2014-08-06 02:30:46,921 WARNI [ckanext.datastore.plugin] Legacy mode active. The sql search will not be available.
2014-08-06 02:30:46,974 DEBUG [ckanext.harvest.model] Harvest tables defined in memory
2014-08-06 02:30:46,979 DEBUG [ckanext.harvest.model] Harvest tables already exist
2014-08-06 02:30:47,016 DEBUG [ckanext.spatial.model.package_extent] Spatial tables defined in memory
2014-08-06 02:30:47,032 DEBUG [ckanext.spatial.model.package_extent] Spatial tables already exist
2014-08-06 02:30:47,078 DEBUG [ckanext.harvest.queue] Fetch queue consumer registered
2014-08-06 02:30:47,080 INFO  [ckanext.harvest.queue] Received harvest object id: 58dc7211-4b27-4b90-85cb-574a1b5d9e8b
2014-08-06 02:30:47,111 DEBUG [ckanext.dcat.harvesters] In DCATHarvester import_stage
2014-08-06 02:30:47,417 INFO  [ckanext.dcat.harvesters] Updated dataset with id a7ad366d-8a96-493e-bc04-fe534b57325a

Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 136, in command
    fetch_callback(consumer, method, header, body)
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 294, in fetch_callback
    fetch_and_import_stages(harvester, obj)
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 317, in fetch_and_import_stages
    obj.save()
  File "/usr/lib/ckan/default/src/ckan/ckan/model/domain_object.py", line 62, in save
    self.add()
  File "/usr/lib/ckan/default/src/ckan/ckan/model/domain_object.py", line 66, in add
    self.Session.add(self)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/scoping.py", line 114, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1252, in add
    self._save_or_update_state(state)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1268, in _save_or_update_state
    self._save_or_update_impl(st_)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1493, in _save_or_update_impl
    self._update_impl(state)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1485, in _update_impl
    self._attach(state)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1513, in _attach
    % (mapperutil.state_str(state), state.key))
sqlalchemy.exc.InvalidRequestError: Can't attach instance <Package at 0x5840950>; another instance with key (<class 'ckan.model.package.Package'>, (u'a7ad366d-8a96-493e-bc04-fe534b57325a',)) is already present in this session.

[urkonn]

en el fetch_consumer.log se puede ver:

    return self.routing_key + ':' + message[self.routing_key]
TypeError: cannot concatenate 'str' and 'NoneType' objects

esto se parece mucho a un bug que tenía ckanext-spacial y que ya está arreglado https://github.com/ckan/ckanext-spatial/issues/70

https://github.com/ckan/ckanext-spatial/blob/74b4141950acb86c53b6c920ca2c613d59b05b2a/ckanext/spatial/harvesters/csw.py#L137

el código de ckanext-dcat es similar https://github.com/ckan/ckanext-dcat/blob/master/ckanext/dcat/harvesters.py#L259