search

OpenDataScotland / the_od_bods

Collating open data from across Scotland
MIT License
21 stars 18 forks source link

Angus council link 99 #201

Closed nutcracker22 closed 1 year ago

nutcracker22 commented 2 years ago

Description

Implemented replace() in ckan.py to replace http: with https: to solve OpenDataScotland/the_od_bods#99 @JackGilmore: did you mean something like this as solution?

Motivation and Context

See OpenDataScotland/the_od_bods#99

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

nutcracker22 commented 2 years ago

Just a side note: when I wanted to write a file to automatically test my changes (=request all the changed links), I recognised that the special characters \r and \n cause a problem when parsing the Angus Council.csv. It is not the first time, that I encounter such a problem. Of course, I could write a work around, but regarding OpenDataScotland/jkan#20, I tend to clean all incoming data right at the beginning.

nutcracker22 commented 2 years ago

okay, when manually testing the URLs, I recognised, that for some the http worked, but the https would not work. So I implemented requests to check for the accessibility. If https is not available, the http URL is kept.

Will try to refactor it tomorrow, to do the https request first (if this request works, the http request can be skipped).

JackGilmore commented 2 years ago

From the looks of it, opendata.angus.gov.uk will only serve over HTTP which is where the mixed content issue is coming from. If you try to force HTTPS then it just doesn't connect at all.

All the GIS-based ones that used data.angus.gov.uk appear to be fine. You can use HTTPS or HTTP redirects to HTTPS anyways from the looks of it. I've dropped an email to the contact link in the about page on their CKAN page to see if they could apply an SSL cert so that HTTPS works on their CKAN instance. If that fails then we might have to look at other solutions such as:

nutcracker22 commented 1 year ago

did you get any response @JackGilmore?

JackGilmore commented 1 year ago

@nutcracker22 I sent through a chaser message via the council's general enquiry last Friday. If no response by this Friday 18th then I've got another contact @watty62 gave me that I can try emailing direct.

JackGilmore commented 1 year ago

Apologies. I kind of dropped the ball on this one by forgetting to send that email. I've sent it now so I propose we wait until the end of next week to see if we get any form of response.

Given the difficulty we've had getting anyone from Angus to speak to us, I'm thinking it's unlikely we'll get any form of a response or action from them on this issue. What I'm thinking is we put some sort of workaround where we display a modal message when a user clicks on a non HTTPS link saying something to the effect of "We can't link you directly to this content because it's not served over HTTPS. Please copy and paste this link into a new browser window: http://example.com"

JackGilmore commented 1 year ago

Yet again I'm lacking in my duties in keeping up with GitHub issues. I did get a reply shortly after emailing my contact in Angus Council. In short, it doesn't look like it's going to be fixed anytime soon.

My thoughts are that we replace the HTTP links with HTTPS where they'll work and then we display a modal message for any leftover HTTP links like I mentioned above. What do you think @nutcracker22?

JackGilmore commented 1 year ago

Closing this PR as unmerged as we've resolved and mitigated this issue with merging in https://github.com/OpenDataScotland/jkan/pull/38