Add workflow to automate updating instances.json

[jarp0l]

This PR adds GitHub actions workflow to automatically update instances.json. It runs on:

• commit to the master branch
• manual trigger
• every 12 hours

[Daniel-Valentine]

Closed per #1.

[spikecodes]

Reopening per https://github.com/libreddit/libreddit-instances/issues/1#issuecomment-1304706533

[spikecodes]

Got this almost working but the generate json script said:

The following instances could not be reached: https://reddi.tk/ https://reddit.dr460nf1r3.org/ https://lr.odyssey346.dev/

despite all 3 instances loading for me. @Daniel-Valentine Any ideas?

[Daniel-Valentine]

@spikecodes: Pull the latest from master. I've ported the changes I did for ferrit-instances, which includes having curl use a different user agent. I ran the script and experienced no problem.

[spikecodes]

Just pulled and reran but got the same issue:

https://github.com/spikecodes/libreddit-instances-workflow/actions/runs/3424539418/jobs/5704326267

[Daniel-Valentine]

@spikecodes: Utterly bizarre. I just ran the script in its latest iteration and observed no issues with any clear-net sites.

Both of those sites are behind Cloudflare. It is possible that Cloudflare is blocking our requests. I'll modify the script to include a more detailed explanation of the failure and that will help us figure out what's going on.

[austinhuang0131]

yeah I know that some Cloudflare configurations could block GitHub requests, see https://github.com/austinhuang0131/joinmatrix/issues/27

[spikecodes]

@Daniel-Valentine Can we have the generate-instances-json.sh script not return an error code (which blocks a GitHub Workflow from completing) when some of the instances are unreachable?

I propose we just don't add those failed instances (only 2 at the moment) to the instance list. If they ask about it, we can explain that they need to change their Cloudflare config to allow requests from GitHub. Additionally, we could have a reminder to do that in the README.

[Daniel-Valentine]

@spikecodes: I can make that change, and I will notify you when it is in.

[spikecodes]

I committed that change (just by changing https://github.com/libreddit/libreddit-instances/blob/master/generate-instances-json.sh#L839 to return 0) but it's not showing in this PR; I guess because jarp0l deleted the branch.

@Daniel-Valentine The one remaining issue is that Tor instances get cleared from the newly-generated instances JSON as the GitHub Action can't make Tor connections. Can we modify the script to accept a command line variable that will "bypass .onion uptime checking" and just include those instances regardless? I tried doing this but got a bit confused with the shell code.

This will be the final step before we can merge this PR and will never have to manually run the instance-updating script again.

[Daniel-Valentine]

@spikecodes: I'm very sorry I'm late to deliver on my promise above (see https://github.com/libreddit/libreddit-instances/pull/3#issuecomment-1336523923) but it's finally in. If you pass the option -e to generate-instances-json.sh, the script will return exit code 0 even when at least one instances fail to respond, though it will still return a non-zero code when the script fails due to a JSON processing error.

As far as keeping the Tor/I2P instances go, use the -I option and specify the original instances.json file as the argument. That will cause the script to import the Tor/I2P instances from instances.json as it existed prior to script invocation and insert them into the JSON it creates, and thus the script will make no connections over Tor or I2P.

All in all, your script invocation should look like this now:

./generate-instances-json.sh -e -i ./instances.txt -I ./instances.json -o ./instances.json

[spikecodes]

Great thanks! Just opened #55 with the proposed changes.