error['message'] is not a string

[gerbyzation]

I've got a setup with ckan 2.7.3 and datapusher 0.0.13.

When pressing 'upload to datastore' on a geojson resource, it updates with the message 'fetching from [..]', and hangs after that. Digging into the datapusher logs the following exception is found:

Fetching from: [redacted]
Error notifying listener
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/apscheduler/scheduler.py", line 239, in _notify_listeners
    cb(event)
  File "/usr/lib/python2.7/site-packages/ckanserviceprovider/web.py", line 184, in job_listener
    db.mark_job_as_errored(job_id, error_object)
  File "/usr/lib/python2.7/site-packages/ckanserviceprovider/db.py", line 419, in mark_job_as_errored
    _update_job(job_id, update_dict)
  File "/usr/lib/python2.7/site-packages/ckanserviceprovider/db.py", line 353, in _update_job
    job_dict["error"] = _validate_error(job_dict["error"])
  File "/usr/lib/python2.7/site-packages/ckanserviceprovider/db.py", line 329, in _validate_error
    "error['message'] must be a string")
InvalidErrorObjectError: error['message'] must be a string
Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at: None)" raised an exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/srv/app/src/datapusher/datapusher/jobs.py", line 412, in push_to_datastore
    raise util.JobError(e)
JobError

The error object triggering this is {u'message': ReadError('Can\'t read Excel file: XLRDError(\'Unsupported format, or corrupt file: Expected BOF record; found \\\'{\\\\n"type"\\\'\',)', <traceback object at 0x7f36a7fc4050>)}, coming from https://github.com/ckan/datapusher/blob/1538e496e5181f94a873f233651c9021f2e676e8/datapusher/jobs.py#L403-L412

The cause of this error is another question, but regardless of that the error handling should not raise an exception itself and consequently CKAN should receive an update with an appropriate message.

Happy to help out on this, but would probably need some guidance from someone who's a bit more familiar with this codebase.

[gerbyzation]

Okay, I was a bit thrown off by the event callback, in this context just calling raise util.JobError(str(e)) would solve the problem. I'm not sure if that would negatively impact other exceptions though.

[timvanoijen]

Any progress or updates on this issue? We are also experiencing the exception "error['message'] must be a string", which is hiding the true cause.

[gerbyzation]

@timvanoijen sorry, I don't really work on CKAN stuff anymore and this is long enough ago I don't remember how we ended up dealing with this.

[alvinChristianto]

I also experienced this issue this week, start with error on ckanserviceprovider

InvalidErrorObjectError: error['message'] must be a string

then followed by error in datapusher

JobError: Did not recognise extension "[typefile]".any: Did not recognise detected MIME type: "[application/...]". any idea ?