New Mutual Aid sea-map - Githubissues

ColmMassey commented 3 years ago

We now need to build a multi source mutual aid sea-map

First new source is the new csv file here https://nextcloud.solidarityeconomy.coop/index.php/s/kQXnTxEDYFTmk8o

So we will need to build new graph for it first.

ColmMassey commented 3 years ago

It should use the following tiles

https://www.thunderforest.com/maps/atlas/

ColmMassey commented 3 years ago

@wu-lee Do you have access to our domains.coop account?

If so, can you set up the mutualaid.coop redirection, with these settings from Chris

If you could point the domain at 81.95.52.65 and let me know when it has been done and I can then generate a new HTTPS certificate and change the WordPress URL, records something like this should do the job:

@ IN A 81.95.52.65 www IN A 81.95.52.65

wu-lee commented 3 years ago

I think you might have given me access once, but I'm not certain about that. More to the point, I don't believe I have the details any more. If you send them over a secure channel I'll add them to the password store whilst doing this.

wu-lee commented 3 years ago

Ok, managed to log in and set these up. They seem to be resolving for me already.

I've also added the login to domains.coop to the password-store repository.

ColmMassey commented 3 years ago

Chris will now set up the certificates for it.

wu-lee commented 3 years ago

Couple questions -

First: is this new mutual aid map in addition to, or replacing, the exisiting mutual aid map?

I ask because I'm wondering whether to create a new directory for the conversion in open-data, or adapt the existing one.

Second: does "multi-source" mean multi-dataset like Newbridge is now? If so, what will the sources be, how should I name them? There will need to be a new data directory (and URI etc.) per source. The CSV above is just one source, in that sense, and I currently don't know what to call it except "mutual-aid" (if replacing) or "mutual-aid2" (if supplementing).

[edit] For now I'm assuming the above CSV is a supplemental source, and creating mutual-aid2, and the names/destinations will need to be retrofitted.

wu-lee commented 3 years ago

Just an observation: although I can use the download link from above to get the CSV file without needing to implement authentication, Nextcloud doesn't seem to set the Etag header (or any related ones) in the response, so like with the LimeSurvey data it can't tell when the file's changed.

wu-lee commented 3 years ago

Given the above, I have a first pass at a working conversion.

Comments:

Item ID 5 has no name, have changes it to be unapproved to avoid errors in the code
For approved, I assume only TRUE is true, anything else is unapproved (and so should be ignored)
I assume facebook is a path on the Facebook.com domain (as usual)

[edit] There's no confirmed name for this, so I won't go to the next step of deploying linked data or a map yet, until we've had a chance to discuss it,

ColmMassey commented 3 years ago

Could we change the covid19 one to covid-mutual-aid.solidarityeconomy.coop and call the new one https://mutual-aid.solidarityeconomy.coop ?

wu-lee commented 3 years ago

We could, so long as no-one is expecting the old one to stay put.

ColmMassey commented 3 years ago

We could, so long as no-one is expecting the old one to stay put.

We never promoted it. This will be the first time it is promoted. We could put a link in the about of the new map, to the big one.

ColmMassey commented 3 years ago

"multi-source" mean multi-dataset like Newbridge is now?

It was originally conceived as needing to be multi-source and may need to be in the future, but for now let's do sincle source.

wu-lee commented 3 years ago

Ok, so there's the new version now hosted here: https://mutual-aid.solidarityeconomy.coop/

Note, the directory is indexing Country, which the current dataset doesn't define. We can either start defining it, or index something else.

Original one will be manually redeployed here when the build has finished (it takes ages). Note, running the sausage machine job cannot yet be switched, because the consequences of the recent breaking changes to open-data need to be resolved first. So if they publish new data we might find it reverts.

ColmMassey commented 3 years ago

Ok, so there's the new version now hosted here: https://mutual-aid.solidarityeconomy.coop/

So what is here now? It looks like the Covid-19 MA groups on SEA Map Version: 0.1.57

wu-lee commented 3 years ago

Whoops, I meant https://dev.mutual-aid.solidarityeconomy.coop/ - I've not monkeyed with the production sites yet.

Note, it also appears that something on the Mutual Aid wiki's data changed and the old data was redeployed, which broke the new version, and will again until I fix the sausage machine job to deploy into the covid-mutual-aid graph. So consider this an incomplete proof of concept.

[edit] I've redeployed it so it works again for now.

wu-lee commented 3 years ago

Ok, so I've moved the old mutual-aid bits and pieces to covid-mutual-aid, so now it builds automatically and deploys to the right place on dev-0 (https://dev.covid-mutual-aid.solidarityeconomy.coop). Point being, it will no longer clobber anything new we deploy on the dev.mutual-aid subdomain.

This involved quite a few steps:

adding the new subdomains dev.covid-mutual-aid. and prod.covid-mutual-aid. and covid-mutual-aid. to our zonefile
renaming the project mutual-aid-project to covid-mutual-aid-project
updating the labels and paths therein accordingly
redeploying it to dev-0
adding covid-mutual-aid to the lod.coop redirector project
redeploying that to dev.lod.coop
redeploying Apache on dev-0 and prod-0 to add the virtual host and updated SSL cert
renaming the mutual-aid directory in the open-data project
updating the labels and paths therein
pushing this change to the dev branch so that the sausage machine on dev-0 picks it up
checking the resulting data is generated ok

I've left the production server stuff as it is, so it will still show the old mutual-aid wiki stuff on mutual-aid. and prod.mutual-aid. subdomains for the time being.

The new dev.covid-mutual-aid. map it looks more or less correct, although not exactly identical to prod.mutual-aid... not sure why, but they aren't running exactly the same version of sea-map, which could be it.

ColmMassey commented 3 years ago

Would you expect the data at dev.lod.coop to be rebuilding now?

wu-lee commented 3 years ago

The data visible at https://dev.covid-mutual-aid.solidarityeconomy.coop/, yes.

ColmMassey commented 3 years ago

and https://dev.mutual-aid.solidarityeconomy.coop/ ?

wu-lee commented 3 years ago

No, https://dev.mutual-aid.solidarityeconomy.coop/ is not managed by the sasuage machine job, it has been manually deployed.

However, https://prod.mutual-aid.solidarityeconomy.coop/ is managed by the sasuage machine job. It has not yet been moved to https://prod.covid-mutual-aid.solidarityeconomy.coop/

wu-lee commented 3 years ago

Note: currently https://dev.mutual-aid.solidarityeconomy.coop/ is using ESSGLOBAL 2.1 and v4 of the standard.csv schema - the latest version which adds countryId and territoryId for the ICA map. This is why I've not added it to the build job, because it's still on the V2a ESSGLOBAL schema (and v3 of the standard.csv schema). Schemas 2.1/4 are not compatible with V2a/3.

But we could make update automatically from the Nextcloud source URL, if I downgrade it to use the old schemas. I don't think it needs anything from the new currently.

The problem is that the open-data project includes the sausage machine implementation, and it is hardwired to support a given output schema. This code is shared by all the datasets. Which means that we can't mix and match schema versions on the same sausage-machine.

So downgrading the schema would be a quick fix to get automatic building working - let me know if you want that.

But I think this will be awkward going forward, as we want to upgrade all the data to the new schema, just not all at once. What I propose to allow this is to split the shared sausage machine code under /tools/se_open_data in open-data into its own repository, like we did with sea-map, and add it back into each data-set as a external library. This means that each data-set will be able to select its own version of the library, and then we will be able to mix and match schema versions.

open-data would remain more-or-less as it is, just lose /tools/se_open_data, and so the deployments on dev-0 and prod-0 could remain more or less as they are too, tracking a branch of the open-data project. When I add this change on the branch, if I do it right, they should just check out the new version and carry on.

If you agree this is a good idea, I'll add this as new ticket to schedule the work.

Aside: our standard.csv schema basically defines the data fields available on initiatives on our maps. As such it's closely coupled to the ESSGLOBAL schemas which define vocabularies and their properties. All of these need a review, as mentioned in https://github.com/SolidarityEconomyAssociation/sea-map/issues/116. And I wonder if the standard.csv schema (currently embedded in the sausage machine code) should be included in ESSGLOBAL - keeping it all versioned together would make sense, I think?

ColmMassey commented 3 years ago

Struggled to follow the order of events you were recommending, but I don't want to do a temporay hack to get Mutual-aid data auto building. How onerous is it to trigger a build manually ATM?

Shall we schedule discussing https://github.com/SolidarityEconomyAssociation/sea-map/issues/116 asap, as I think that is holding you back?

wu-lee commented 3 years ago

How onerous is it to trigger a build manually ATM?

It's conceptually fairly simple to rebuild the data and deploy it. The main problem with Mutual Aid is the build is very slow, it takes in the order of hours I think. (At least on my desktop.)

ColmMassey commented 3 years ago

Crossed wires here. When I say mutual-aid I am referring to the tiny Nextcloud file sourced data. I'll say covid-mutual-aid to refer to the big dataset.

wu-lee commented 3 years ago

Crossed wires here. When I say mutual-aid I am referring to the tiny Nextcloud file sourced data. I'll say covid-mutual-aid to refer to the big dataset.

Ah ok, that's quick because not much data.

Struggled to follow the order of events you were recommending, but I don't want to do a temporay hack

Ok. Casting any temporary hack aside, then, does it seem ok to spend the order of hours to a day decoupling the various open-data data sets so they don't have to output the same target schema? This means we can switch the target ESSGLOBAL version one data set at a time in future.

Shall we schedule discussing SolidarityEconomyAssociation/sea-map#116 asap, as I think that is holding you back?

Sure. It is probably needed for all the ESSGLOBAL related changes to go forward.

wu-lee commented 3 years ago

Note, I've created a new GitHub project called mutual-aid-project, for the non-covid mutual aid data described here. Can we move this issue there @ColmMassey ? (I'm not sure I can do that.)

[edit: found the option to do it so I took the liberty of transferring it.]

wu-lee commented 3 years ago

Ok. Casting any temporary hack aside, then, does it seem ok to spend the order of hours to a day decoupling the various open-data data sets

As discussed, I've started investigating this, seems feasible, and I've created a ticket to cover this work: https://github.com/SolidarityEconomyAssociation/technology-and-infrastructure/issues/61

wu-lee commented 3 years ago

@ColmMassey this is implemented, using ESSGLOBAL2.1, and merged into thedev branch, along with EG2.1 versions of covid-mutual-aid, ICA, and ICA Youth Network.

The one snag is that the data for this map is invalid, so it won't build currently. The reason is, the last line has no name, which is a mandatory field. Suggested fix below (remove the TRUE value in the approved column), but we could also just remove it entirely, or add a name. I don't think I have access, so perhaps you could do this @ColmMassey?

I need to investigate ways of getting feedback when the data is invalid like this.

--- original.csv    2021-05-01 17:57:56.391974620 +0100
+++ original.csv    2021-05-05 16:23:08.501754946 +0100
@@ -3,4 +3,4 @@
 2,Climate Cafe Hub,https://transitionliverpool.org/,,,53.38046;-2.9615,TRUE,sdfsdf
 3,Wirral Environmental Network,https://wirralenvironmentalnetwork.org.uk/,,,,TRUE,zxcxvcxvzxvxv
 4,Dale Farm Trust,https://www.dalefarmtrust.org/,,,53.41654;-3.02857,,
-5,,,,,53.33244;-3.11396,TRUE,xcvxvxvxcvxvxv
+5,,,,,53.33244;-3.11396,,xcvxvxvxcvxvxv

ColmMassey commented 3 years ago

I am confused about which spreadsheet we are using.

For some reason I have been populating this one...

https://nextcloud.solidarityeconomy.coop/index.php/s/YDctt5nbP4e7Rx7

ColmMassey commented 3 years ago

The link at the top of this issue,

https://nextcloud.solidarityeconomy.coop/index.php/s/kQXnTxEDYFTmk8o

doesn't seem to editable via the browser.

ColmMassey commented 3 years ago

https://nextcloud.solidarityeconomy.coop/index.php/s/YDctt5nbP4e7Rx7 , the one I have been editing seems to point to Mutual-Aid-Mini.ods in the ..CodeOperatives/MutualAid folder

ColmMassey commented 3 years ago

NextcloudDownloadTest.csv is the one you are currently linked to in a NextCloud folder called Public

ColmMassey commented 3 years ago

I have overwritten NextcloudDownloadTest.csv with other files, content. What were the restrictions on using spreadsheets in Nextcloud for this purpose?

ColmMassey commented 3 years ago

Did you find what was stopping the map from updating? Just checking I am looking at the right url for the manually updated spreadsheet

https://dev.mutual-aid.solidarityeconomy.coop/ ?

ColmMassey commented 3 years ago

@wu-lee What is holding this back form working?

wu-lee commented 3 years ago

There's a problem with the mutual aid scripts, but so far I've been focusing on getting the email notification of this problem working rather than fixing the problem itself, because I'd like to see evidence that the failure notifications arrive. This has involved an email exchange with WebArchitects.

There's now a reply from WebArchitects which is saying we can use their mail system to relay the notifications, I just need to make this work (possibly with a bit of help from them). There are other alternatives, but they tend to involve trying to use gratis-tier services on things like Mailgun, which tends to have rate limiting, or setting up our own mail server, which is time consuming and requires ongoing maintenance.

If this doesn't work out soon though, I'll just fix the problem and break it again later to test the notifications.

ColmMassey commented 3 years ago

I haven't been able to follow the mail server notification issues, but I think we may need to put that aside and fix the map updating problem now.

wu-lee commented 3 years ago

Ok, I've fixed mutual aid and the job is running ok now.

There were a couple of problems, one trivial, one a bit of a headache.

The trivial problem was just a configuration error - some settings from the config I use for testing locally had leaked in and the deployment was trying to upload to dev-0 without a password. Easy fix.

The not-so-easy problem was that the new Nextcloud spreadsheet URL does not work like the old one. The old one gets a CSV file, presumably because the actual file stored is a CSV. The new one is an Open-Office .ods file, and does not download as a CSV, but a binary Open-Office blob. I thought I'd checked this was working during our meeting last week, but I must have made a mistake.

There seems to be no option for this to be converted to CSV, so I needed to modify the config file and add a new dependency on a Ruby module which can read .ods format files (Roo). Cue other problems with bugs in Roo and installation errors.

But, these are squished now, and the conversion works satisfactorily. You can see the resulting map here:

https://dev.mutual-aid.solidarityeconomy.coop/

Of course, there may be problems to solve for that too now. For instance, they all seem to be labelled "Co-operative"...

ColmMassey commented 3 years ago

Nice. Great that we now have ods support. Leave display of org type for the moment. Add description & twitter.

wu-lee commented 3 years ago

Regarding the mail issues, I concluded that WebArch's mail server isn't really equipped to make it easy to use it as an SMTP gateway for notifications from the sausage machine. Therefore for now, having done a bit pf RTFM I've managed to use my own mail serve rat noodlefactory.co.uk as an SMTP gateway.

This isn't ideal since it's my infrastructure, not SEA's, but works for now. Possibly a better option is to pay a few dollars per month for a pay-as-you-grow Mailgun account, specifically about 80 cents per 1000 emails. In the worst case and it sends once every ten minutes, there'll be 62431 = ~4.5k emails per server, so about $9/month for both servers. But the mails should drop to zero now it's mailing only on an error.

ColmMassey commented 3 years ago

Possibly a better option is to pay a few dollars per month for a pay-as-you-grow Mailgun account,

Happy to do that under Digital Commons. No point setting up on SEA account and then have the fun of transferring.

wu-lee commented 3 years ago

Ok, I've filed off some rough edges to do with sea-map with things which were added for the ICA (mainly that a map site which didn't use vocabs would make it throw an error), and bumped the dependency of the mutual-aid-project map site to a newly released sea-map@1.0.0 (ICA-style vocab support, ESSGLOBAL 2.1, WebPack, plus John's dialog improvements)

Also v2.1.0 of se-open-data - with some Twitter normalisation code.

Finally, redeployed this map using all of the above.

https://dev.mutual-aid.solidarityeconomy.coop/

Meawhile, seems that the data has been modified, and there is only one (approved) initiative, "NYC Mutual Aid". Knowing that, this seems to be working.

So overall, mostly all done (except I see back at the beginning this was intended to be a multi-source map?)

ColmMassey commented 3 years ago

The Twitter handle is being stored as a url, and then in the dialog the url is being appended to the url!

DigitalCommons / mutual-aid-project

New Mutual Aid sea-map #1