search

ckan / datapusher

A standalone web service that pushes data files from a CKAN site resources into its DataStore
GNU Affero General Public License v3.0
77 stars 155 forks source link

Problem with Inf in decimal field #101

Open chrism0dwk opened 8 years ago

chrism0dwk commented 8 years ago

Hi,

I'm experiencing an issue with datapusher failing to upload a CSV file into my CKAN datastore when certain columns contain 'Inf' values. Details below.

I have verified that messytables is detecting the column type correctly (Decimal) from within Python, so I'm assuming that the problem is the Python to PostgreSQL step (i.e. the error I'm seeing is being returned by the database, rather than being a Python error code).

Is this behaviour currently expected? If so, I'll have a look at the source and see if I can suggest a fix.

Thanks,

Chris

CKAN version 2.5.2 on Ubuntu 14.04 installed from packages

Error message:

Error: The data was invalid (for example: a numeric value is out of range or was inserted into a text field). HTTP status code: 409 Response: {"help": "http:///api/3/action/help_show?name=datastore_create", "success": false, "error": {"message": "The data was invalid (for example: a numeric value is out of rang... Requested URL: http:///api/3/action/datastore_create

amercader commented 8 years ago

I tried locally and the Postgres type that is passed for a field containing Inf is numeric. According to Postgres docs you can only use this on Floating point types like real. The mapping on DataPusher is defined as Decimal -> numeric, so I'm not sure about the implications of changing this or potential workarounds.

Hope this helps.