search

asc-csa / ckanext-asc-csa

📈 Extension CKAN pour le portail de données et information ouvertes de l'ASC | 📈CKAN Extension for the CSA open data and information portal
https://donnees-data.asc-csa.gc.ca/dataset
Other
7 stars 2 forks source link

Enabling the Data Explorer page in CKAN 2.9 #126

Closed nfee006 closed 3 years ago

nfee006 commented 3 years ago

The best I've been able to get is this.

image

That's using a publicly available dataset with DataProxy. I still haven't been able to get datapusher/datastore working properly in 2.9.

Have you had any luck with this @coleweusthem?

nfee006 commented 3 years ago

This may be related. I got this traceback when I created a new resource with the datapusher running. Seems that something went wrong when it tried to push the csv from the public file to the datastore.

image image

coleweusthem commented 3 years ago

can you give me the steps you took to get this?

nfee006 commented 3 years ago

  1. Create a new dataset. http://127.0.0.1:5000/dataset/new image

  2. Fill in and submit the dataset creation form (button at the bottom of the page). image

  3. Fill in and submit the resource creation form (button at the bottom of the page). Set the URL to a public dataset such as https://open.canada.ca/data/dataset/f781b1ff-1205-4e94-93c8-dbb3b8e5973c/resource/f2c7ba28-ed9a-4feb-9bab-9168ccd0e153/download/presentation-csv.csv.
    image

That traceback appears when submitting the form in step 3.

Edit: @coleweusthem, I just went through the steps again and did not get an error this time... It still didn't actually work (ie. create the Data Explorer page views or push to the datastore) though. Not sure what's going on with that.

nfee006 commented 3 years ago

@coleweusthem I tried it again just now (same steps as above) and the error came back. Not clear why it's inconsistently doing so.

image

coleweusthem commented 3 years ago

@nfee006, For the data explorer at the top of this issue, did you try a dataset that works for sure in the old version? I've tried the one you attached above and best I got was a loading gif that keeps spinning for the data explorer, which is definitely different from what you see.

nfee006 commented 3 years ago

@coleweusthem yeah, the dataset works for sure on the old one. I just tried it again with https://open.canada.ca/data/dataset/0176458c-553b-48b4-a5e2-492022c81e85/resource/b335e213-16e9-4755-8125-a22b55319cb5/download/go_canada_carte_data_fr.csv (which I also know works) and it was the same.

You might already know this, but a quick summary of how the data is served on that page:

  1. If the resource has been successfully pushed to the datastore postgres database ( using datapusher) then it'll take the data from there. Theoretically, if you have datapusher running, it should automatically push it to the datastore when you add the data URL to the resource.
  2. If the resource is not in the datastore, it'll try to use DataProxy or ResourceProxy instead. This doesn't seem to be working properly (which isn't necessarily a huge deal because we'd rather use the datastore)

When I add the url to the resource, it isn't pushing it to the datastore (I have datapusher running though). When I click on the data explorer page, I get that warning that the Dataproxy has failed to retrieve the dataset. If I go in and manually push the resource to the datastore (example url: http://127.0.0.1:5000/dataset/bfb2331e-6d68-41c0-8cf4-c2e9c4d68eb7/resource_data/7c0baaaf-ff13-45d2-bfd2-7e1cbe6cf629), it does so successfully. However, when I go back to the data explorer page, I get that Werkzeug routing error that I posted above.

coleweusthem commented 3 years ago

@nfee006 , I'm not sure exactly what is going on, but is it possible that the canada.ca IT staff blocked services like the datapusher from pulling from their servers? I downloaded the CSV to my machine, hosted it from my mac using MAMP, and tried adding it to CKAN, when I did that it seems to have worked when I pressed the upload to datastore button.

Screen Shot 2021-06-10 at 3 30 28 PM Screen Shot 2021-06-10 at 3 30 42 PM
nfee006 commented 3 years ago

That would be surprising but they did upgrade their CKAN yesterday - I can ask and I'll try it again on my side. I would try the same with a CSV from another site. Example: https://dd.weather.gc.ca/air_quality/aqhi/atl/forecast/model/csv/2021060212_AQHI_ATL_AQFM.csv

coleweusthem commented 3 years ago

ok I'll give that one a try next

nfee006 commented 3 years ago

Can confirm that this one works on my VM https://open.canada.ca/data/dataset/ca983b71-df22-431a-9d5f-94e77b835dd5/resource/f5329912-ed23-4f39-9c7d-086e003e4d26/download/funded-open-access-publications_2011-2017.csv

That's good though! Not getting that error in this case. I'll continue testing it.

nfee006 commented 3 years ago

@coleweusthem There's also the issue of translating the data explorer page

image

In 2.8 this was done by editing that recline.js file that I sent you. 2.9 has a new version of recline though - it might support i18n out of the box, not sure. Feel free to chose the approach that you want to take, but we do need it translated. Let me know if you need assistance with the French text (though you can take it from CKAN 2.8 as well).

coleweusthem commented 3 years ago

@nfee006, do you have a way of maintaining the javascript file? or is it just something you change when you deploy CKAN

coleweusthem commented 3 years ago

recline.zip

nfee006 commented 3 years ago

This was a kinda hack-y thing I did because we couldn't find any work-arounds. Are you saying that that .js file will work for 2.9?

coleweusthem commented 3 years ago

Yes it should, and I think the hacky way is the only way to do it currently. The text is all inline in the code to begin with. I don't think it supports translations of any kind.

nfee006 commented 3 years ago

A few things left to translate

image

looks like this in 2.8

image

It should all need to be changed in recline.js. It's possible that the recline file is out of date. Did you make any changes to it or could I grab it from prod and replace it in the repo?

coleweusthem commented 3 years ago

I had made changes to the the one that was installed with ckan. They might have changed how that part works from the older version. You could grab it from prod but I'm not sure it would work correctly.

nfee006 commented 3 years ago

Gotcha - ok. Are you able to look into this then please?

nfee006 commented 3 years ago

Also, when testing this page I found the following.

Normally, when you click on the arrow next to "Go to this resource" it gives you an option of a few file formats. That doesn't seem to work anymore on my side.

2.8 image

2.8 image

There's also no API button but that might be a vanilla 2.9 thing? Still, I imagine it might be related.

coleweusthem commented 3 years ago

@nfee006 Can you give me some class names or info on the Data API button, it's missing and I'm trying to track down what controls it by running searches in the code base

nfee006 commented 3 years ago

Like this? image

coleweusthem commented 3 years ago

yes thanks

nfee006 commented 3 years ago

Here's an example of what it looks like in 2.9 btw https://demo.ckan.org/dataset/sample-dataset-1/resource/e687245d-7835-44b0-8ed3-0827de123895

coleweusthem commented 3 years ago

thanks, weird that its not showing

nfee006 commented 3 years ago

@coleweusthem: has the "preview experimental" button disappeared for you as well? It should be visible for all CSV/TXT files when logged in as admin and for all of them where preview = true when logged out.

My VM right now when logged in as admin:

image

Should look like this:

image

coleweusthem commented 3 years ago
Screen Shot 2021-06-23 at 2 07 12 PM

Think it might be something specific to your end, mine looks like this.

nfee006 commented 3 years ago

Noted, I'll poke around.

nfee006 commented 3 years ago

I got the button back, but only for new datasets. It must have something to do with how I ported over the metadata from prod. I'll try a different method.

coleweusthem commented 3 years ago

In my research it only shows for datasets that have been saved to the datastore

nfee006 commented 3 years ago

Hmm - should be to use data proxy if something isn't in the datastore though. I don't think I've seen the preview button disappear when logged in as admin yet.

nfee006 commented 3 years ago

@coleweusthem - testing out the data download button on the data explorer page. Was there somewhere to specify a filename/extension? In CKAN 2.8 the downloaded file has the appropriate extension, but in the new version it's just an extension-less file. Otherwise, it seems to work.

WAVE is giving an empty button error image

and W3 has errors image

There's also the issue of the translations image

coleweusthem commented 3 years ago

@nfee006, I just pushed a fix for the translations, when I use the download button I'm getting files with extensions. And I'll look at that w3 error

coleweusthem commented 3 years ago

@nfee006, I fixed the first error, I'm not getting any of the others. Can you make sure you've pulled down the latest changes and check again? maybe send me the specific resource you are looking at?

nfee006 commented 3 years ago

I confirm that my local instance is up to date with the python3_updates branch.

Good for the translations when I moved the recline.js file to the right place.

image

I'm working with a test dataset for all of these since my preview buttons still aren't working otherwise (I think I have the fix for that but it might be a little risky so it'll wait until Monday). The resource URL I'm using is https://open.canada.ca/data/dataset/f781b1ff-1205-4e94-93c8-dbb3b8e5973c/resource/f2c7ba28-ed9a-4feb-9bab-9168ccd0e153/download/presentation-csv.csv.

File extensions: still having the same issue in English and French. I'm using firefox if that makes a difference? I was trying to download the resource as a csv but got a file without any extension.

image

W3 errors are still there but only in French.

image

WAVE is still flagging the empty button error from my last comment as well.

coleweusthem commented 3 years ago

@nfee006, I think I've got the iframe issue, still can't replicate the file without an extension, and my wave isn't showing an empty button. we may need to set up a call to sort those 2 out.

nfee006 commented 3 years ago

Works fine in the python3_updates branch.