Continue to standardize process of creating a spreadsheet-based new layer

[jywarren]

We have layers that drive from google spreadsheets -- like the PFAS layer:

https://github.com/publiclab/leaflet-environmental-layers/blob/main/src/pfasLayer.js

we should improve the workflow for this, as in #135 - for specifically Google Spreadsheet-based layers

@kevinzluo made an awesome generic utility for this at https://github.com/publiclab/leaflet-environmental-layers/pull/105/ which we could build on!

Example spreadsheet we could link to in the docs: https://docs.google.com/spreadsheets/d/1biyMqC4pk4pezG6EuJ6oSXSMBDlZFxfpleqv5BlTSaM/edit#gid=0

The format of the spreadsheet should be:

Title	Notes	Latitude	Longitude
My title	the notes for this entry's popup	44	-71

It'd look like this:

[Sagarpreet]

I think we can make a form (a small button with popup maybe and add to publiclab.org somewhere) and ask people to fill it with:

1. google sheet
2. column names
3. lat/ lng columns On submitting the form we can use github api's (fire github API directly from browser) and make an issue in LEL repo (maybe a first timer issue?)

Once an issue is made, we can write an interactive script (similar to create release script here: https://github.com/publiclab/PublicLab.Editor/pull/617) which will ask for information like google sheet url, columns, etc in terminal and later make all the code changes automatically using some template files and will use Google Spreadsheet-based utility layer and run grunt command also to bundle new changes.

Now after running this script, the first-timer contributor just has to make PR. What do you think @jywarren @ebarry ?

[ebarry]

I think i understand, and i think i like it. @bhamster07, please follow along here :) I'm going to try to recap:

For the person contributing data, they would have to:

1. Create a googlespreadsheet with the columns described above by Jeff
2. Click a button on publiclab.org that will open a simple widget described by Sagarpreet, and fill in its fields
3. Click submit

(THEN ROBOTS DO STUFF, including creating a first-timers-only issue in this repo with next steps)

Then, a new contributor would:

1. comment on the issue that they are going to take it up
2. run a script and do the stuff Sagarpreet describes above
3. Make a PR

Then the regular processes of our website maintainers take the next steps: review the PR, merge, republish the website, and the layer will then be visible in the layer palette on any map on the website, including on /map.

Is this correct?

[sagarpreet-chadha]

Absolutely correct @ebarry 👍 💯

[ebarry]

Just circling back to say that @bhamster07 who would be a data contributor has said that "Workflow definitely seems doable, straightforward and easy :thumbsup:"

[Sagarpreet]

@ebarry @jywarren ...few points:

1. For creating first-timers-only issue from plots2 we can use either of these:

   • To make request from browser itself: https://www.jsdelivr.com/package/npm/@octokit/rest
   • To make request from rails server: http://octokit.github.io/octokit.rb/Octokit/Client/Issues.html#create_issue-instance_method

2. We will need to create a new github account, maybe: publiclab-bot? and use its personal access token (only non-admin permission token can be generated which should be safe).

[jywarren]

This sounds great, i wonder if the level of integration is a bit deep in that perhaps a direct request from rails to the Octokit API would be a brittle thing to maintain - API integrations are tough to test for. What if we made a more lightweight workflow which involved an issue template, which then could be manually copied into a FTO template with ALL_CAPS_REPLACEABLE_TEXT strings corresponding to, for example, the spreadsheet URL, the name of the layer, the color of the markers?

This could be automated in a later step but it would be more robust to failures - it wouldn't depend on the Octokit/Rails integration and if a script or bot failed, it's pretty easy to continue doing it manually, so the breakage wouldn't be a show-stopper.

Open to other ideas - but just thinking about who maintains the bot, you know? ❤️ 🤖

what do you think?

[sagarpreet-chadha]

Correct @jywarren doing manually is straightforward, but one will need to make FTO issue manually from template issue everytime new layer is requested.

I was reading octokit api to create issue, it is simply a wrapper over github api and needs title, description, and gh token as params. So this should not break easily.

For the script, which will do all changes in the repo automatically for adding new layer that I agree is more prone to break if we change folder structure or maybe some package upgrade.

Anyways as a backup we should definitely document how to do all this manually 👍 💯

[sagarpreet-chadha]

So there is no bot actually in above flow, we are just making a new github account and using its personal access token, which we will pass to octokit wrapper function to create FTO issue 😄

[jywarren]

Let's start by making the template, and go through it manually once, then we can automate?

But would this Octokit script be part of the plots2 repo?

[jywarren]

I guess i might prefer if the script could be in JS in this repo, do we have the option for JS vs. Ruby w/ Octokit?

[sagarpreet-chadha]

Okay sure we can first document everything that needs to be done if done manually, i.e from format of excel sheet to making FTO to making layer and PR.

Yes as mentioned here https://github.com/publiclab/leaflet-environmental-layers/issues/488#issuecomment-719529090 We can do this in ruby or JS.

So for now we should not go ahead with either of the 2 scripts? which are:

1. To create FTO issue using gh api’s
2. To create files for making LEL layer

[jywarren]

Since (2) will make the system work (even in a reduced way) but (1) won't by itself, i guess let's start with (2)!

[daemon1024]

Hi, Is there any progress on this? This seems pretty interesting to me and I would love to collaborate/work on it.

Regarding creating issues and automating things using github api, we don't necessarily need to have a personal access token, we can use the OAuth API or use Github Apps directly ( an example would be how welcome-bot ) works. Though this seems to be on a later priority.

Edit: I realised this is part of a SoC project so I will approach this through the proper channel.

[daemon1024]

I was brainstorming on this issue and I have a nice way to bridge the gap between opening the issue and running the script.

So, we can trigger the running of script using the GitHub actions,

I have created an example workflow here https://github.com/daemon1024/issue-action-script-run. This just logs the issue description in the action.

This wouldn't need a server dependency while also maintaining the manual ecosystem if this fails. Let me know if this could be an addition the process would be happy to propose this in detail. cc @jywarren @sagarpreet-chadha

Update :

Workflow