search

GSA / datagov-wptheme

Data.gov WordPress Theme (obsolete)
https://www.data.gov
Other
1.88k stars 411 forks source link

Data previews not working. #458

Closed rossjones closed 9 years ago

rossjones commented 10 years ago

It looks like data previews are not currently working. There's still an ajax request happening to jsonpdataproxy.appspot.com (the centralised data preview server) and recline.js is being loaded but previews are not showing up on the pages like http://catalog.data.gov/dataset/healthcare-associated-infections/resource/4c72e08b-078f-4d97-a94f-684f14d339f9 or http://catalog.data.gov/dataset/cms-medicare-and-medicaid-ehr-incentive-program-electronic-health-record-products-used-for-80f66/resource/3c9a3d91-0330-45fc-bd17-eb26f65a56bd

Is this intentionally disabled?

philipashlock commented 10 years ago

The first URL doesn't link to an XLS (or requires authentication) and we'll try to address that kind of issue with #55. I think the second one should be working, so we'll investigate more. It's over 50mb, so it might be too big for the proxy.

Here are a few examples where it's working for CSV and XLS files:

http://catalog.data.gov/dataset/multiemployer-pension-plans/resource/f063d6eb-d50c-45ef-aca7-fcb6e31295be http://catalog.data.gov/dataset/fdic-failed-bank-list/resource/608b288c-46bb-493c-a8ff-a2a0d05b51a0

Unfortunately it seems as if previews fail more often than not. I think mostly this should be addressed with #55 but we'll try to investigate how much it's happening for other reasons

rossjones commented 10 years ago

Yes, they both work for me.

The proxy runs on appengine and so suffers the 32Mb(?) download limit, and inflexible error messages :( - it also fails as often as it succeeds. At datapress.io we are working on a flask-based replacement (with XLS sheet chooser) so it may be that that will solve a lot of the problems as you'll be able to host versions locally. Obviously it still needs to download the resource (unless you have it archived) but it does cache.

philipashlock commented 9 years ago

Seems like there might be a few separate issues here. Some of these might be related to existing issues, but just to list them here for now:

ydave-reisys commented 9 years ago

Datapreview issue is mostly on xlsx format. According to http://docs.ckan.org/en/ckan-2.1/data-viewer.html#viewing-structured-data-the-data-explorer dataviewer is not supported for xlsx format.

https://github.com/okfn/dataproxy - this does mention that it supports xlsx but it doesn’t work. We get data transformation error. xlsx url can be tested on this site: http://jsonpdataproxy.appspot.com/static/demo.html

max-results parameter can be used for large data.

Other format preview works fine in Firefox browser.

ydave-reisys commented 9 years ago

Updated code to hide preview for xlsx format. If there is mismatch of format in url and mimetype/format mentioned in additional info section then format mentioned in additional info section will be considered. Also if url is broken then preview will not be shown. In scenario where url is xlsx and format mentioned is xls preview will be disabled.

We can make change in this logic to always consider format from url but that will need us to revise the logic, during harvesting, to guess what format the resource is. The impact of this will be on lot of things and thus will require more time.

kvuppala commented 9 years ago

Preview are working correctly now for the allowed formats like (CSV/XLS) both http/https versions of the site.