Keep Tor onion service up-to-date

[andrewdavidwong]

We have a Tor hidden service mirror, thanks to @Nukama. However, it has fallen woefully behind the updates to the canonical website. (As of this writing, most pages on it are inaccessible.) Is there any way to automate the task of keeping it up-to-date? If not, perhaps we shouldn't list it in the footer of the site (at least not without some sort of disclaimer).

For reference, here's the URI: http://qubesosmamapaxpa.onion/

[marmarek]

@Nukama, I can setup some hook to call when the repository is updated. On your side this would require exposing some address (either clearnet or tor service), which will call _utils/update-submodules (or similar). The script would take care of git tag verification. That exposed hook would not need to parse any parameters, just fetch updates from github repo.

Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

[andrewdavidwong]

We probably shouldn't wait for a reply from @Nukama on this.

[qubenix]

Maybe host the hidden service on the same server as the clearnet site? This will dramatically simplify this process and make an easier workflow for the dev team (if @Nukama will give the private key of course).

[marmarek]

Currently clearnet site is hosted on github pages, so not exactly possible to host hidden service from there. But cloning github pages (jekyll) is very simple.

[fortasse]

I can probably host this along side Whonix's hidden services. However, if you want to keep the same .onion domain, you'll have to get me the private key somehow. Otherwise, we can have another subdomain on kkkkkkkkkk63ava6.onion.

If it's just a matter of pulling a git repo every so often, that should be pretty easy to do.

[marmarek]

I can probably host this along side Whonix's hidden services. However, if you want to keep the same .onion domain, you'll have to get me the private key somehow. Otherwise, we can have another subdomain on kkkkkkkkkk63ava6.onion.

Unfortunately, we don't have private key for the old address :/ @andrewdavidwong do you have any preference as for the address - from the list I've sent you?

If it's just a matter of pulling a git repo every so often, that should be pretty easy to do.

I can also setup github hook for the repository - it will sent a simple POST request to configured URL. Possible problem here: it must be clearnet URL, onion addresses are not supported. If really needed, I could proxy it myself, but not sure if worth the effort.

[fortasse]

Our hidden service and clearnet sites are hosted on the same machine, so having the hook tie back to a clearnet address should be perfectly fine. I don't have too much experience with webhooks, however.

[marmarek]

It's easy: https://github.com/marmarek/signature-checker/blob/master/github-webhook - expose as CGI. If unique script could be used for this particular purpose - probably you don't need to parse POST data at all. Some documentation: https://developer.github.com/v3/repos/hooks/#receiving-webhooks And actual example event for git push: https://developer.github.com/v3/activity/events/types/#pushevent

[andrewdavidwong]

Unfortunately, we don't have private key for the old address :/ @andrewdavidwong do you have any preference as for the address - from the list I've sent you?

No preference. There weren't any outstanding ones. Here were the slightly recognizable addresses:

qubesoseggc4nxja.onion
qubesos4rrrrz6n4.onion
qubesoseaaeexn7o.onion
qubesosxuuxxnkzb.onion
qubesosangieqolx.onion
qubesosnojpylaqe.onion
qubesosxydie7non.onion
qubesosgtgxgxdot.onion
qubesosprrwr4z3h.onion
qubesoscirect6nz.onion

IMO, using a subdomain on kkkkkkkkkk63ava6.onion would also be fine.

[marmarek]

I think I like qubesos4rrrrz6n4.onion from this list (for me the easiest one to remember). @mfc?

[mfc]

yep qubesos4rrrrz6n4.onion sounds good.

[fortasse]

Cool. If you want to email me the private key, my contact info is here: https://www.whonix.org/wiki/contact#For_Website_Issues

[fortasse]

I have qubesos4rrrrz6n4.onion up and running. @marmarek: I sent you an email to get the webhook / automation side part set up as well.

[marmarek]

Ok, webhook enabled. @andrewdavidwong do you have some pending edits to test if automation works?

[fortasse]

For full disclosure, the reason I emailed the webhook link directly is to prevent it from getting crawled by search engines and regenerating the site unnecessarily. If you email me directly and provide a half-decent reason for needing the endpoint, I'll happily provide it.

[fortasse]

If there are no editorial changes to make, adding a comment to robots.txt would likely work.

[marmarek]

If you want to prevent spurious calls, github provide a mechanism for verifying if request really come from github - X-Hub-Signature header: https://developer.github.com/v3/repos/hooks/#webhook-headers But just for anti-crawlers, you can just check some simple property of received data - like if that is a POST request (very rare for crawlers), if X-GitHub-Event header is present etc.

Anyway, just keeping URL non-public is also some solution.

[fortasse]

Aah, good things to point out. Once we make sure everything's working I may go back and improve the actual webhook receiver, it's a bit inelegant at the moment.

[marmarek]

If there are no editorial changes to make, adding a comment to robots.txt would likely work.

I'd like to check also submodules update. Just pushed some minor edit.

[fortasse]

Doesn't look like it worked. @marmarek, would you mind hopping in #whonix on OFTC to help me get this debugged?

[fortasse]

I've redone the webhook to essentially use a modified version of the post-receive hook in the readme here. That should deal with the submodules properly.

[fortasse]

@marmarek joined me in IRC, and I believe everything is working now. I will make a note to clean up my implementation and potentially add it to _utils in the main repo.

[andrewdavidwong]

Added a footer link to the new hidden service. Thanks, @fortasse!

[anonmos1]

Can we have https://www.qubes-os.org/qubes-issues/ working on the onion too?

[andrewdavidwong]

Can we have https://www.qubes-os.org/qubes-issues/ working on the onion too?

That one's hosted from a different repo. The original intention was for it not to share the qubes-os.org domain at all, but because of the way the website is set up, there's an automatic redirect from https://qubesos.github.io/qubes-issues/ to https://www.qubes-os.org/qubes-issues/. Not sure how much trouble it'd be.

CC @fortasse @marmarek

[fortasse]

If we're cool with it just living at qubesos4rrrrz6n4.onion/qubes-issues/ it shouldn't be too much trouble. We can use the existing webhook, I can just update the script to update both repos / folders.

[andrewdavidwong]

@fortasse: Sounds good to me. Thanks!

[fortasse]

@andrewdavidwong: http://qubesos4rrrrz6n4.onion/qubes-issues/ is up. Do you want to confirm the webhook works? It's exactly the same as the old webhook, just with a few more steps to pull and build https://github.com/QubesOS/qubes-issues/tree/gh-pages. I assume it will work fine, but more testing is never a bad thing. :)

[andrewdavidwong]

@fortasse Ok, I've pushed several commits to the gh-pages branch. They're not reflected on the onion version yet, but it might still be too soon.

[fortasse]

This might be a stupid question, but did you configure this repo to hit the same webhook as the main website repo? The last hit to the webhook was at 27/Apr/2017:17:49:44 +0200 according to logs.

[andrewdavidwong]

No, I've never done that before. (I think @marmarek did it for the main website repo.) How would I do that? Feel free to point me to documentation.

[fortasse]

https://developer.github.com/webhooks/creating/

TLDR: To set up a repository webhook on GitHub, head over to the Settings page of your repository, and click on Webhooks & services.

You should clone whatever settings are used for the QubesOS/qubes-doc repo, I'm not sure what the exact settings are on the github-side.

[andrewdavidwong]

Oh, I don't have admin access to either repo, so I can't even view the Settings page. I think @marmarek will have to do it.

[fortasse]

Alright. I just hit the webhook manually, can you confirm the changes are reflected on qubesos4rrrrz6n4.onion/qubes-issues/ as expected? If hitting it manually works, it should have no issues once the webhook config gets updated on Github.

[fortasse]

I'm 95% sure it should work when the repo config is updated. Let me know if it still doesn't work once the webhook is configured on the github-side.

[andrewdavidwong]

Yep, looks good. Thanks, @fortasse!

[marmarek]

Webhook added. But there is some problem - it failed with "Service timeout", and also the last one for main repo failed at 2017-04-27 17:47:51 (the last one).

[fortasse]

Marek Marczykowski-Górecki:

Webhook added. But there is some problem - it failed with "Service timeout", and also the last one for main repo failed at 2017-04-27 17:47:51 (the last one).

That might be an issue on the github side, and it happens occasionally? I see 200s for the webhook from the github hookshot in the logs, so I think it works? I do see the HTTP 499 at 28/Apr/2017:09:59:50 +0200, but everything after that is 200s.

I believe github's behavior is to retry webhooks that fail a given number of times.

[anonmos1]

onion doesn't have the latest announcement

[fortasse]

anonmos1:

onion doesn't have the latest announcement

I don't see any POSTs to the webhook from Github. Did someone change the config? The last log line is from 03/May/2017:15:10:01 +0200.

I'll update it manually in the meantime.

[marmarek]

According to github - this is also when the last POST was sent (successfully). And it is after adding an announcement. So, something is wrong with the update script...

[fortasse]

Shoot, you're right - I had a typo in one of the directories. It should be fixed now. My bad, thanks for pointing it out @anonmos1!

[anonmos1]

Unfortunately, the onion is not up to date

[fortasse]

Just updated it manually, and made a couple tweaks to the update process. Hopefully that will prevent this from happening again.

Sorry for the trouble, and thanks for letting me know.

(Also, it looks like reply-by-email is broken on Github. If this double posts, sorry.)

[andrewdavidwong]

Thanks for taking care of it, @fortasse!

[anonmos1]

I still cannot see the latest announcements

[fortasse]

Hm, apparently the --incremental flag for Jekyll doesn't work like I thought. I've removed it, and confirmed the updates are now there. It seems like I was pulling the files from Git properly, but not generating the site completely...

Can you confirm @anonmos1?

[anonmos1]

Yes, all seems to be good

[anonmos1]

Last 4 announcements are missing

[anonmos1]

@fortasse