[CWM] Update Co-ops UK Open Data Map

[ColmDC]

Track time against Cooperative World Map Project

Is your feature request related to a problem? Please describe. The Co-op UK Open Data map is not automatically updated and hasn't been manually updated since summer 2023.

Describe the solution you'd like

• [x] Update the map to use ~Jan~ June 2024 data
• [x] Update the prod map to ~3.1.4~ 3.1.7
• [ ] ~Move from coopsuk.solidarityeconomy.coop to maps.coop/coops-uk (Inform DotCoop that the map is being moved so they can change on their site.~

[wu-lee]

The data has been updated (manually) - no obvious problems except that it's large and slow. CSV published at http://dev.data.solidarityeconomy.coop/coops-uk/standard.csv

The map published here is already at 3.1.4

http://dev.coopsuk.solidarityeconomy.coop/

Problems arrived when trying to view in the map: the data seems to have crossed some sort of size threshold and triggered a bug, which may be in the PapaParse CSV library. See https://github.com/DigitalCommons/mykomap/issues/229

[ColmDC]

Blocked until we have a solution to https://github.com/DigitalCommons/mykomap/issues/229

[wu-lee]

Ok. Now that that solution to large CSVs has been implemented by adjusting the CORS config of dev-1, I have tested the CUK development map (link as above). I can confirm it downloads the whole dataset correctly - no errors, and the final-row organisation is visible.

So this can now progress.

This issue doesn't appear to request that we add CUK's data to be automatically processed. But if we do, I anticipate a problem in that they use different file names for each new release, at least as listed here:

https://www.uk.coop/resources/open-data

So we might need to ask them to create a URL which points at the latest version always.

[ColmDC]

Need to find out what Agile is going to do with the data streams.

[ColmDC]

Need to find out what Agile is going to do with the data streams.

No response from them, so let's just keep to using the monthly data

[ColmDC]

No response from them, so let's just keep to using the monthly data

Just got confirmation that there will be no new API for co-ops uk data, so we just keep with the old open data csv in old location.

[ColmDC]

Not sure why this has been bumped to Review.

[wu-lee]

It is because I mentioned in the last delivery meeting that I'd deployed the Co-ops UK data as part of the work on the demo-merge-map plus-plus: https://github.com/DigitalCommons/mykomap/issues/249

But I forgot to update this issue at the time.

Specifically, the June 2024 data was deployed on 4th June.

This should show in the CUK maps immediatel here:

https://dev.coopsuk.solidarityeconomy.coop/

The production version has not yet been updated though:

https://coopsuk.solidarityeconomy.coop/

It can't be totally automated and scheduled yet, because we don't have a reliably consistent URL to download the data from.

[ColmDC]

How much effort is it to move to maps.coop? So maps.coop/coops-uk

It can't be totally automated and scheduled yet, because we don't have a reliably consistent URL to download the data from. The url does change but looks predictable. (something for the future...)

[ColmDC]

The about text needs updating

The data was last updated for Q2 2023. -> The data was last updated for June 2024.

Also remove the lasrt sentence which is not needed, as there is a link to the license in the first paragraph.

[wu-lee]

How much effort is it to move to maps.coop? So maps.coop/coops-uk

It's a matter of just deploying it there, but we really need to move all of them and avoid getting ourselves confused by having a mixture on each server and no clear way of knowing which is the "right" one. This is an ongoing job. Currently the "go-to" server for this is still dev-1/prod-1 - except for the limequery maps, which have been moved as part of https://github.com/DigitalCommons/technology-and-infrastructure/issues/106.

Moving these maps is the topic of https://github.com/DigitalCommons/technology-and-infrastructure/issues/109 specifically, and more generally https://github.com/DigitalCommons/technology-and-infrastructure/issues/107.

[wu-lee]

The about text needs updating

Will do this. Although perhaps we shouldn't rely on updating this text manually every time it's redeployed, as that's prone to human error (whoever does it has to remember this extra step for this map).

[Edit: moreover, the "about" text is a property of the map deployment, and which CUK dataset is in use isn't. Usually, the data for our maps has a separate deployment process, which is automated and can't update the "about text" in the consuming map application. Perhaps the map should be able to show some metadata obtained from the dataset - then we don't have to tweak the map deployment just because the data was redeployed? This would be a new mykomap feature, however.]

[wu-lee]

I notice there's no Clockify project defined for this issue - perhaps there should be?

[wu-lee]

The about text needs updating

Done on dev-1. (Production left as-is)

[ColmDC]

I notice there's no Clockify project defined for this issue - perhaps there should be?

Just track against CWM

[ColmDC]

It's a matter of just deploying it there, but we really need to move all of them and avoid getting ourselves confused by having a mixture on each server and no clear way of knowing which is the "right" one.

Just thought it might be worth doing while working on the map, but note your point that it would be more efficient to move them all together.

[ColmDC]

[Edit: moreover, the "about" text is a property of the map deployment, and which CUK dataset is in use isn't. Usually, the data for our maps has a separate deployment process, which is automated and can't update the "about text" in the consuming map application. Perhaps the map should be able to show some metadata obtained from the dataset - then we don't have to tweak the map deployment just because the data was redeployed? This would be a new mykomap feature, however.]

Sounds like a feature.

[ColmDC]

avoid getting ourselves confused by having a mixture on each server and no clear way of knowing which is the "right" one.

Shipshape is I guess how we have done this before and will still do for a while.

[ColmDC]

So let's just push to prod and we can close this ticket.

[wu-lee]

Ok, we need to update the data-feed production branch too, for this. Bringing it more in line with the data-feed dev branch - at least to get the coopuk changes currently being used on that.

But this also brings in some dotcoop data-set changes and ica data-set changes (neither of which seem to have officially gone live on production).

I don't think this will be a problem, but would need to go a bit slowly - step by step, checking neither ICA or DotCoop maps break.

[wu-lee]

Actually it looks like the production branch of the ICA data hasn't been moved the the automatic data feed. It's been a while, but I don't remember a final sign off on that.. The map hasn't changed either (still on an ancient version of MM). So I might need to hot-fix the production branch so it doesn't rebuild the ICA until we get that.

[wu-lee]

Ok: updated coops-uk map on prod-1 now deployed. Both prod and dev should be on MM 3.1.7.

The data isn't yet deployed - it's a manual process still, but leaving it to run overnight, it's very slow to complete.

Production branch of data-feed has been hot-fixed to

• not build the ICA,
• nor owned-by-oxford
  • we need to check deploying to the production Murmurations servers is ok
  • also it was broken by the API authentication changes anyway
  • this can be resolved when owned-by-oxford gets a proper deploy

[wu-lee]

Ok, coops-uk data now deployed.

[ColmDC]

So as we are not moving to maps.coop until we are migtrating the others, we can close this ticket. Can you repeat the hotfix notes in the respective tickets for those delayed releases. (https://github.com/DigitalCommons/ica-project/issues/37) (https://github.com/DigitalCommons/owned-by-oxford-project/issues/33) ?

[wu-lee]

Done.