Open florianm opened 9 years ago
Also getting this error constantly, though my CKAN instance version is 2.3
, and harvesting seems to be seriously broken.
My config settings are:
{
"remote_groups": "only_local",
"force_all": true,
"user": "harvest",
"api_version": 1
}
And a local group catalogue is present.
You probably know this, but just in case: when you get an ORM object (e.g. with model.Package.get(id) or model.Session.query(model.Package).first()) then you can use it until you do a model.Session.commit()/close()/remove(). After that you get a DetachedInstanceError if you try and use the ORM object again. That is the most likely cause.
I suggest you put in a breakpoint or use prints to find out where this is happening and fix it.
Seems like adding 'defer_commit': True
to the context dictionary in a custom harvester prevents the DetachedInstanceError
from being raised. Can someone confirm this?
Dave, thanks for the info - didn't know that. As I was using a vanilla ckan harvester it didn't know I had to intervene at that level. Let me double check that it wasn't due to some mods on my end.
Raphael, thanks I'll try that out as soon as I can, gonna be early next week! On 08/09/2015 7:42 pm, "Raphael Stolt" notifications@github.com wrote:
Seems like adding 'defer_commit': True to the context dictionary prevents the DetachedInstanceError from being raised. Can someone confirm this?
— Reply to this email directly or view it on GitHub https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138497118 .
Please everyone, let's help each other more - if no-one tries to fix problems like this (or pay someone to do it), and contributes back as PRs, nothing will improve. There is no magic central funding for CKAN that guarantees things work.
Deferring the commit sounds like a bad solution that complicates things. It will break-down at some point in a confusing way. I'd encourage you to take the time to find the actual place where the commit is done and the object is used after it.
On 8 September 2015 at 12:17, Florian Mayer notifications@github.com wrote:
Dave, thanks for the info - didn't know that. As I was using a vanilla ckan harvester it didn't know I had to intervene at that level. Let me double check that it wasn't due to some mods on my end.
Raphael, thanks I'll try that out as soon as I can, gonna be early next week! On 08/09/2015 7:42 pm, "Raphael Stolt" notifications@github.com wrote:
Seems like adding 'defer_commit': True to the context dictionary prevents the DetachedInstanceError from being raised. Can someone confirm this?
— Reply to this email directly or view it on GitHub < https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138497118> .
— Reply to this email directly or view it on GitHub https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138521331 .
Nope, not deferring, only on holidays this week and banned from coding by executive order of wife. Will work on it first thing when I'm back - hope that's enough to avoid the wrath of ckan. On 08/09/2015 9:32 pm, "David Read" notifications@github.com wrote:
Please everyone, let's help each other more - if no-one tries to fix problems like this (or pay someone to do it), and contributes back as PRs, nothing will improve. There is no magic central funding for CKAN that guarantees things work.
Deferring the commit sounds like a bad solution that complicates things. It will break-down at some point in a confusing way. I'd encourage you to take the time to find the actual place where the commit is done and the object is used after it.
On 8 September 2015 at 12:17, Florian Mayer notifications@github.com wrote:
Dave, thanks for the info - didn't know that. As I was using a vanilla ckan harvester it didn't know I had to intervene at that level. Let me double check that it wasn't due to some mods on my end.
Raphael, thanks I'll try that out as soon as I can, gonna be early next week! On 08/09/2015 7:42 pm, "Raphael Stolt" notifications@github.com wrote:
Seems like adding 'defer_commit': True to the context dictionary prevents the DetachedInstanceError from being raised. Can someone confirm this?
— Reply to this email directly or view it on GitHub <
https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138497118>
.
— Reply to this email directly or view it on GitHub < https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138521331> .
— Reply to this email directly or view it on GitHub https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138526001 .
Wait I think I misunderstood what Dave meant with "deferring". Nevertheless the WiFi up here is too shaky to ssh into my ckan. On 08/09/2015 9:42 pm, "Florian May" florian.wendelin.mayer@gmail.com wrote:
Nope, not deferring, only on holidays this week and banned from coding by executive order of wife. Will work on it first thing when I'm back - hope that's enough to avoid the wrath of ckan. On 08/09/2015 9:32 pm, "David Read" notifications@github.com wrote:
Please everyone, let's help each other more - if no-one tries to fix problems like this (or pay someone to do it), and contributes back as PRs, nothing will improve. There is no magic central funding for CKAN that guarantees things work.
Deferring the commit sounds like a bad solution that complicates things. It will break-down at some point in a confusing way. I'd encourage you to take the time to find the actual place where the commit is done and the object is used after it.
On 8 September 2015 at 12:17, Florian Mayer notifications@github.com wrote:
Dave, thanks for the info - didn't know that. As I was using a vanilla ckan harvester it didn't know I had to intervene at that level. Let me double check that it wasn't due to some mods on my end.
Raphael, thanks I'll try that out as soon as I can, gonna be early next week! On 08/09/2015 7:42 pm, "Raphael Stolt" notifications@github.com wrote:
Seems like adding 'defer_commit': True to the context dictionary prevents the DetachedInstanceError from being raised. Can someone confirm this?
— Reply to this email directly or view it on GitHub <
https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138497118
.
— Reply to this email directly or view it on GitHub < https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138521331
.
— Reply to this email directly or view it on GitHub https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138526001 .
Sorry Florian, I overreacted. I just want to guard against people thinking this stuff should work flawlessly. Of course it's good to ask around like you have. I just want to encourage everyone to get their hands dirty and fix things with PRs too, and I know you are one of the best examples of that.
On 8 September 2015 at 12:46, Florian Mayer notifications@github.com wrote:
Wait I think I misunderstood what Dave meant with "deferring". Nevertheless the WiFi up here is too shaky to ssh into my ckan. On 08/09/2015 9:42 pm, "Florian May" florian.wendelin.mayer@gmail.com wrote:
Nope, not deferring, only on holidays this week and banned from coding by executive order of wife. Will work on it first thing when I'm back - hope that's enough to avoid the wrath of ckan. On 08/09/2015 9:32 pm, "David Read" notifications@github.com wrote:
Please everyone, let's help each other more - if no-one tries to fix problems like this (or pay someone to do it), and contributes back as PRs, nothing will improve. There is no magic central funding for CKAN that guarantees things work.
Deferring the commit sounds like a bad solution that complicates things. It will break-down at some point in a confusing way. I'd encourage you to take the time to find the actual place where the commit is done and the object is used after it.
On 8 September 2015 at 12:17, Florian Mayer notifications@github.com wrote:
Dave, thanks for the info - didn't know that. As I was using a vanilla ckan harvester it didn't know I had to intervene at that level. Let me double check that it wasn't due to some mods on my end.
Raphael, thanks I'll try that out as soon as I can, gonna be early next week! On 08/09/2015 7:42 pm, "Raphael Stolt" notifications@github.com wrote:
Seems like adding 'defer_commit': True to the context dictionary prevents the DetachedInstanceError from being raised. Can someone confirm this?
— Reply to this email directly or view it on GitHub <
https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138497118
.
— Reply to this email directly or view it on GitHub <
https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138521331
.
— Reply to this email directly or view it on GitHub < https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138526001> .
— Reply to this email directly or view it on GitHub https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138529275 .
BTW I've just added a testing framework for harvesters, which might help you identify the problem, if it is to do with your specific harvester. And if you can write a failing test it helps other people reproduce and help with the problem. See this example for the CKAN harvester:
@davidread thanks for the testing framework, I'll give it a whirl!
FYI my particular harvesting requirements ended up being highly customised and co-evolving with the harvesting process, so I scripted the process in the first instance, here's the code for reference. I'm breaking away from the ckanext-harvesting paradigm in two incompatible ways:
Interesting point about ckanext-scheming and harvesting - I've not used scheming yet but I guess it means adapting whatever harvester for that.
The WMS/WFS thing looks annoying to deal with.
I quickly looked at your 'harvest' scripts and I guess you could simply drop them into a harvester gather_stage and they'd run fine, upsert and all. Then the harvest framework would give you a web front-end for viewing and running them, could schedule runs etc. but you'd lose having them in a python notebook which is easier to toy with when necessary. (The harvest framework assumes you'd create datasets across a single organization, but if you hid or ignored that field I don't think there'd be a problem.)
Encountered the same error. My CKAN is version 2.4.0 running on an ubuntu, not running any customisations. Strangely, harvesting worked for several months, until just now. I'll list what I did, in case it might help find the origin problem. I'm not a Python developer, so I can't trace this down myself.
harvester run
script.Below is the error stacktrace. This error occured when I started harvesting, and now occurs whenever I try to start the fetch_consumer script.
(default)dieter@dieter-VB-CKAN:~$ paster --plugin=ckanext-harvest harvester fetch_consumer --config=/etc/ckan/default/production.ini
2015-11-09 14:32:14,214 DEBUG [ckanext.harvest.model] Harvest tables defined in memory
2015-11-09 14:32:14,216 DEBUG [ckanext.harvest.model] Harvest tables already exist
2015-11-09 14:32:14,329 DEBUG [ckanext.harvest.queue] Fetch queue consumer registered
2015-11-09 14:32:14,330 INFO [ckanext.harvest.queue] Received harvest object id: 373e3b8b-14c9-4abd-978b-ebd3544b06d1
2015-11-09 14:32:14,360 DEBUG [ckanext.dcat.harvesters.rdf] In DCATRDFHarvester import_stage
2015-11-09 14:32:14,483 DEBUG [ckanext.harvest.harvesters.base] Update validation Error: {'Resources': 'Package resource(s) invalid'}
2015-11-09 14:32:14,488 INFO [ckanext.harvest.queue] Received harvest object id: 95004674-f3c2-4681-930b-d2365c9cac0d
2015-11-09 14:32:14,501 DEBUG [ckanext.dcat.harvesters.rdf] In DCATRDFHarvester import_stage
Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 144, in command
fetch_callback(consumer, method, header, body)
File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 305, in fetch_callback
fetch_and_import_stages(harvester, obj)
File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 322, in fetch_and_import_stages
success_import = harvester.import_stage(obj)
File "/usr/lib/ckan/default/src/ckanext-dcat/ckanext/dcat/harvesters/rdf.py", line 273, in import_stage
existing_dataset = self._get_existing_dataset(harvest_object.guid)
File "/usr/lib/ckan/default/src/ckanext-dcat/ckanext/dcat/harvesters/rdf.py", line 82, in _get_existing_dataset
return p.toolkit.get_action('package_show')({}, {'id': datasets[0][0]})
File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 429, in wrapped
result = _action(context, data_dict, **kw)
File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/get.py", line 927, in package_show
_check_access('package_show', context, data_dict)
File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 296, in check_access
data_dict)
File "/usr/lib/ckan/default/src/ckan/ckan/authz.py", line 153, in is_authorized
user = _get_user(username)
File "/usr/lib/ckan/default/src/ckan/ckan/authz.py", line 118, in _get_user
if c.userobj and c.userobj.name == username:
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 233, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 577, in get
value = callable_(state, passive)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line 422, in __call__
self.manager.deferred_scalar_loader(self, toload)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 563, in load_scalar_attributes
(state_str(state)))
sqlalchemy.orm.exc.DetachedInstanceError: Instance <User at 0x5370450> is not bound to a Session; attribute refresh operation cannot proceed
Note that the invalid resource error is probably related to one of my harvesters using an incorrect DCAT feed.
When going down the rabbit hole to fix or better said avoid this error, I noticed when disabling the core CKAN recline_view
plugin or replacing it with recline_grid_view
the error went away. Don't really know how these intertwine but worked for me.
How exactly do you disable/replace that plugin?
For reference, my extentions are, as listed on <host>/api/util/status
: "stats", "text_view", "image_view", "recline_view", "harvest", "ckan_harvester", "dcat", "dcat_rdf_harvester", "dcat_json_harvester", "dcat_json_interface"
Further update: Since I couldn't immediately figure out how to fix that error, I reverted my VM that was running CKAN. It was an almost fresh install. I retried to add the harvesting URL, and immediately encountered the same issue again. Perhaps it has to do with the DCAT that is being read?
I also noticed that the harvester fetch_consumer
command keeps crashing as long as the harvest job is running (as seen on <host>/harvest/admin/<job-name>
. Pressing Clear
on that page allows the harvester fetch_consumer
to start again.
Simply by modifiying the plugins list (i.e. ckan.plugins) in /etc/ckan/default/production.ini
.
Final update: I have a situation where I can recreate the error. It is related to the dcat file (my dcat file is missing some necessairy attributes), but I can't narrow it down. I managed to find a single dataset entry in the DCAT that crashed it, only to work it when I reinitialized the VM.
When starting from a completely fresh CKAN installation, I can't reproduce the error, so I guess it has to do with the data that is already present in the DB.
However, I'm not sure how to further investigate this error. My Python-fu is limited, and I have no idea how to debug something that interacts between 3 python processes. If someone gives me some pointers (within reasonable time of this post), I can gather more data.
@DieterDePaepe Would you providing the DCAT file? Either with a link or putting it in a pastebin, so I can try creating a test?
The stacktrace shows the problem occurs when the harvester's import stage, when tries to package_show the pre-existing dataset and that tries to access c.userobj. Now c.userobj is initialized in BaseController, but I'm struggling to see how that would have been run by a harvester. I can't get it to initialize running the existing DCAT harvester tests.
BTW did disabling the recline_view plugin make the error go away?
@DieterDePaepe You could also try to see what is initializing c.userobj, by inserting this breakpoint into ckan/ckan/lib/base.py around line 281:
# model directly.
+ import pdb; pdb.set_trace()
if c.user and not c.userobj:
and then run the fetch_consumer. Hopefully it should give you a prompt and you can type bt
to give you the stack. Could you try that and let me know what the stack shows?
DCAT feed that causes crash for me can be found at http://pastebin.com/9j9amRxg Note the missing DownloadURL in each dataset.
As I mentioned before I tried to narrow down the DCAT to produce the same error. I ended up with the 2nd listed dataset (about crime). But since this smaller DCAT didn't cause an error when I tried again, I'm not sure it's strictly related to that one.
Haven't tried yet to disable the recline_view plugin. Will try that and the debug command soon.
I tried disabling the recline_view plugin and tried replacing it with recline_grid_view. After restarting apache and reloading the harvesters, the error was still present.
Adding the python debugger: after adding it, the ckan website gives an internal error, the logs mention a BdbQuit error, which according to google occurs when the debugger cannot get i/o for interaction. So that makes sense and shouldn't interfere with the debugging of the harvester. However, when running the run command, I get the same DetachedInstanceError as before, no debugger. Perhaps this is because I am running a paster command?
The commands I use to run harvesting are (each in a different console that I keep open):
(default)dieter@dieter-VB-CKAN:/usr/lib/ckan/default/src/ckanext-dcat$ paster --plugin=ckanext-harvest harvester fetch_consumer --config=/etc/ckan/default/production.ini
(default)dieter@dieter-VB-CKAN:~$ paster --plugin=ckanext-harvest harvester gather_consumer --config=/etc/ckan/default/production.ini
(default)dieter@dieter-VB-CKAN:~$ paster --plugin=ckanext-harvest harvester run --config=/etc/ckan/default/production.ini
Any update on this issue?
We start to see this errors on one of our servers (it's actually on our test server, fortunately the production works). But they use the same data and have the same configuration.
The Instance <User at 0x3410450> is not bound to a Session
error in a custom harvester in the import_stage, in a call to organization_show, to check if an organization exists. It's the first time the context is used. I thought it might be related to https://github.com/ckan/ckanext-harvest/commit/5e1512f717abfb9ad9fc6af897bcf55b86da83b9, but copying the context didn't help.
Does this give any clues? https://github.com/ckan/ckan/pull/1714#issuecomment-43745098
Sorry, it's a long read but very thorough and it fixed a similar problem I had once with spatial harvesters, so might be worth checking.
@amercader after an extended debugging session, we were finally able to pin down this issue. It was a very unfortunate chain of circumstances that led to this error:
resource_update
makes some nasty stuff to the given context and we didn't make new copies for all our callsresource_update
and our custom code was less than idealOur custom harvester has the requirement to not update the package metadata, but only updated the ressources (because it's possible that the package metadata is changed manually after the harvester runs). Therefore we make calls to resource_create
and resource_update
, without caring about the package itself.
This failed for most of our packages, i.e. we got in the except block with a KeyError for the key 'resources'. After some looking around, we found the code responsible to this KeyError:
try:
context['defer_commit'] = True
context['use_cache'] = False
_get_action('package_update')(context, pkg_dict)
context.pop('defer_commit')
except ValidationError, e:
errors = e.error_dict['resources'][-1]
raise ValidationError(errors)
Here comes the interesting part for the DetachedInstanceError. Because we were guilty of not create a new context for each of our calls, this code changed our context. i.e. _get_action()
would raise an exception, leave context['defer_commit'] = True
in the context for all subsequent calls.
To add more confusion, the raised exception does not contain a resources
key on the error_dict attribute, therefore raising a KeyError. I don't know anything about that. But in the end the actual ValidationError was that we used a key on the package in as extra fields (schema error). This was because those datasets were not changed since CKAN 2.2 and therefore were still saved in the old format.
We ended up adding a little script to "migrate" the old packages to get rid of the extras. This leads to the situation, that we don't get a ValidationError anymore, and therefore everything runs smoothly.
The problem, that the code in resource_update
and resource_create
could cause a corrupt context and/or throw KeyErrors in an except block is still not solved. I don't know exactly, how this code should behave. But most probaly the context.pop('defer_commit')
should be in a finally
block, so that the context is restored, even if there was a ValidationError. I'm open to create PR for this, but this is probably not the repository to have this discussion anyway.
I just wanted to leave this here, in case someone needs some more hints to debug this.
@metaodi Really glad to hear that you were successful after what it sounds like an intense debugging session :)
Legacy harvesters might have partial blame, but the KeyError
on resource_update
update is terrible and several people have been bitten by it. There is a PR to provide a better handling, it's just lacking tests. I'll try to find time to write one or feel free to jump on it if you have some: https://github.com/ckan/ckan/pull/3027
I just installed the new release of CKAN (2.7.3) and encountered same error
INFO [ckanext.harvest.queue] Received harvest object id: fd026470-dd37-408f-9b57-e819c52f70bf
DEBUG [ckanext.harvest.harvesters.ckanharvester] In CKANHarvester import_stage
DEBUG [ckanext.harvest.model] Instance <User at 0x7f70731823d0> is not bound to a Session; attribute refresh operation cannot proceed
I followed the whole process of installing ckan, datastore, datapusher and harvester. I am trying to harvest: https://data.smartdublin.ie/. I tested different sources and the same error occurs. Please advise me what I did wrong or what shall I do to make everything work properly.
When I had to migrate data into CKAN, I gave up on the harvester, and used the CKAN API from an iPython notebook: https://github.com/datawagovau/harvesters (same link as in my comment above)
Harvesters are great when running as a scheduled task, but introduce one extra level of complexity which can conflict with other customisations (e.g. ckanext-scheming overriding "extra" fields). Simply whacking the API with a wrench until the data goes in always works. While this might be a workaround in some cases, I support Dave's advice of actually trying to find a fix.
@MaciejJanowski you can discount datapusher and datastore. The problem you're seeing is with ckanharvester.
A bit about your context might help us think about this:
paster harvester run
. Have you tried testing it with paster harvester run_test
? If so, does that work?@davidread
@MaciejJanowski thanks for this. All sounds standard enough.
The tests pass, so I I guess the next step is for you or another developer to investigate why it happens and/or reproduce it in a test. You could use print statements or pdb to find out when the User object is instantiated, when its tried to be used and it's found to be detached, and then track down when it gets detached. At least you're seeing it happen every time, which helps(!)
I'm hitting the same problem while using the CSW harvester, using
FWIW, here's what I observe and how I fixed it, although I'm far from understanding what's exactly happening.
The root cause is a harvested package that fails to validate, causing an error and a rollback of the session (see https://github.com/ckan/ckan/blob/e838541d/ckan/logic/action/create.py#L175). This apparently causes the userobj in ckan.common.c to become detached. As a result, when processing the next harvested object, a call to is_authorized (in my case to get the license list : https://github.com/ckan/ckanext-spatial/blob/f805d412/ckanext/spatial/harvesters/base.py#L294) chokes on the userobj being detached (https://github.com/ckan/ckan/blob/e838541d/ckan/authz.py#L119).
I fixed it by checking if the userobj is still attached :
if c.userobj and c.userobj.Session.object_session(c.userobj) and c.userobj.name == username:
return c.userobj
That's not a satisfying fix; i'd rather understand why this userobj gets detached in the first place or whether it should be re-attached, but that's as far as I can go.
When harvesting a CKAN instance (source install v2.5.0a as of July 2015) into another CKAN instance (datacats using latest master, v2.5.0a) with a custom ckanext-scheming dataset schema, I'm running into DetachedInstanceError indicating that the db conection via SQLAlchemy changes sessions.
My config:
I've created a sysadmin user called "harvest", same error as with my own sysadmin account.
I haven't seen a DetachedInstanceError since mentioned in the dev meeting archives a year ago. Has anyone seen this error lately, or is harvesting only broken on my end?