search

whatwg / misc-server

Miscellaneous resources for the servers hosting *.whatwg.org domains
Other
23 stars 16 forks source link

Transition whatwg.org to new hosting #7

Closed foolip closed 6 years ago

foolip commented 7 years ago

To improve our uptime and allow for more admins, we are moving the hosting of whatwg.org from DreamHost to DigitalOcean. These are the domains:

foolip commented 7 years ago

Separate from the hosting question, these are static sites that could be hosted inside https://github.com/whatwg/whatwg.org/:

@annevk?

annevk commented 7 years ago

I'd like @sideshowbarker's opinion on everything help.whatwg.org.

I'd be okay with merging whatwg/resources.whatwg.org into whatwg/whatwg.org at some point, though leaving it alone for now seems fine too? Seems somewhat more complicated if we want to keep the history intact.

For the two others I created https://github.com/whatwg/whatwg.org/pull/52.

annevk commented 7 years ago

As a next step I think we should try to move everything that's not *.spec.whatwg (so we will move spec.whatwg), blog.whatwg, or wiki.whatwg. Then we could move blog.whatwg and wiki.whatwg and maybe html.whatwg (it seems like we'll continue to have a need to execute scripts so we might want to make a longer term plan first), and last all the specs (that are all intertwined I think).

foolip commented 7 years ago

Yeah, let's leave resources.whatwg.org alone, because it's used by HTML it might get more traffic, so might not want it deployed in the same place. Will you put images.whatwg.org and n.whatwg.org into the repo though? We can set up the deploy for that before DNS is changed.

foolip commented 7 years ago

Oops, that's https://github.com/whatwg/whatwg.org/pull/52

sideshowbarker commented 7 years ago

help.whatwg.org can just be redirected

foolip commented 7 years ago

resources.whatwg.org is a tricky one, because it's used by all specs. If it's not on the same IP and using the same cert as the specs, then load time will be less awesome than it could be. Maybe we should try hosting everything static on a single machine, and just make it beefier?

annevk commented 7 years ago

I think we should host all static things on the same server instance. But note that any performance benefits won't come until Let's Encrypt actually offers multi-domain wildcard certificates.

foolip commented 7 years ago

We could also request a single cert for all the domains on a single machine.

foolip commented 7 years ago

The downside is that if the DNS of any one of those domains changes, I think certbot renew would start failing.

foolip commented 7 years ago

The current setup for spec.whatwg.org (nested inside whatwg.org source-wise) leads to fun: https://whatwg.org/specs/web-forms/current-work/ https://spec.whatwg.org/web-forms/current-work/

These both work, but the latter looks like an accident. I suggest that we just let those URLs go 404, and preserve only the original URLs.

annevk commented 7 years ago

Yes, we should do that.

foolip commented 7 years ago

I'm extracting out spec.whatwg.org now.

foolip commented 7 years ago

Handing html-differences in https://github.com/whatwg/html-differences/pull/5

foolip commented 7 years ago

I've added lists.whatwg.org to the list, it's currently broken by HSTS, and it would be nice to restore it somehow.

annevk commented 7 years ago

@foolip I suspect we can't get ownership of that until we have DNS control since it's highly likely a name that's tightly coupled to DreamHost's managed mailing list service. (When we take that over we should make sure that whatwg@lists.whatwg.org still works. Same for whatwg@whatwg.org.)

foolip commented 7 years ago

Right, so the challenge is how to get the archives from DreamHost and, presumably, to keep updating them if we're not shutting down all of the lists.

annevk commented 7 years ago

Well, the main list we'd end up keeping is also archived at https://lists.w3.org/Archives/Public/public-whatwg-archive/. If we can it might be better to just find a mapping from the old URL to those URLs and redirect everything there. The alternative would be to install the same mailing list software (unless we want to deal with local redirects).

foolip commented 7 years ago

That would work, if https://lists.w3.org/Archives/Public/public-whatwg-archive/ contains everything that our own archives do. Still probably a lot of work.

annevk commented 7 years ago

It should be complete, but yes.

foolip commented 7 years ago

@domenic, is any action needed for build.whatwg.org? At the very least we shouldn't mess up its DNS in eventual DNS handover.

domenic commented 7 years ago

It might be a good idea for me to move that all from my personal AWS to one of the droplets. It'd also be good to switch to LetsEncrypt.

If you're up for it, I'd appreciate the help doing so; the readme at https://github.com/domenic/wattsi-server should have most of the info. Then you could switch the DNS to point to the droplet and I could shut down my server completely.

Eventually we probably want to move that to a whatwg/build.whatwg.org repository.

foolip commented 7 years ago

I've added it to the list so that it's not forgotten.

foolip commented 7 years ago

What's left blocking HTML is now print.pdf. Should try to make https://github.com/izh1979/makepdf part of the Travis build. Thanks @izh1979 for putting it together!

foolip commented 7 years ago

So I'm thinking maybe we could just build/install https://github.com/izh1979/makepdf on Travis for each commit?

foolip commented 7 years ago

The other alternative would be to make it part of build.whatwg.org, but then we'd have to worry about malicious input, and not just processing what gets merged into the spec.

annevk commented 7 years ago

The more we can outsource to Travis the better I think.

domenic commented 7 years ago

Yes, putting it in the Docker stuff for CI build is my plan. It sounds like I should work on that sooner rather than later.

foolip commented 7 years ago

Yeah, otherwise we'll have to either run PHP on the machine serving HTML, or on some other machine that accepts the PDF and deploys it.

Getting the pdfbuild repo working on Travis isn't trivial, but is probably possible in the end.

izh1979 commented 7 years ago

Or you can rewrite the script in some other language. ;-) It's not big, nor complex. And it's easier when you have working variant. ;-)

snuggs commented 7 years ago

@izh1979 @foolip where would said script be located? Or is this related to https://github.com/whatwg/html-build/issues/120#issuecomment-323981383 & https://github.com/whatwg/html-build/pull/122/

Lastly, what is http://sgr-a.ru/h/whatwgpdf.php ?

P.S. @annevk 👍 for PDF'ing the standards.

Thanks in advance!

annevk commented 7 years ago

@snuggs the script used for the HTML Standard is hosted at https://github.com/izh1979/makepdf and that .php page runs it.

foolip commented 7 years ago

Well, this is amusing, while trying to figure out how to do this, I came across https://www.princexml.com/forum/topic/2666/using-better-compression-and-pdf-size-optimization which is @izh1979 asking about compressing PDFs :)

izh1979 commented 7 years ago

Yeah, I've generating PDFs for WHATWG HTML spec for about of 3 years. ;-)

foolip commented 6 years ago

All that remains here now is lists.whatwg.org, which is tied together with the DNS. Somebody needs to learn how email actually works to do this job, maybe we should liaison with the IETF :)

foolip commented 6 years ago

An update on lists.whatwg.org. https://packages.debian.org/sid/mailman3-suite looked very promising, but is only in Debian unstable, and installation didn't actually work out of the box on a fresh Debian unstable, neither an (upgraded) DigitalOcean droplet or in a debian:unstable Docker image. (I think it failed to create/start a postgres db, and is probably easy-ish to fix.)

Because Debian unstable also doesn't get security updates in the same way, it's probably best to follow http://docs.mailman3.org/en/latest/prodsetup.html or similar, and https://github.com/whatwg/misc-server/commit/213da82f2ddc0d97d7fff5987d56947ea97e0b7a is what I tried so far. With some more tinkering it's probably possible to make that work.

I haven't tried https://github.com/maxking/docker-mailman yet because it seems somewhat implausible to me that the total complexity will be lower, one still has to have a database and web server separately, and we don't want archives, so it'd pretty much just give us mailman3-core and the django stuff.

On setting up an email server (not just mailman), something very discouraging at https://www.digitalocean.com/community/tutorials/how-to-set-up-a-postfix-e-mail-server-with-dovecot:

Postfix has several hundred configuration parameters. If you want to administer a mail server that reliably delivers business requirements to a sizable organization, you should make yourself intimate with all of them (man 5 postconf). This tutorial will not be enough, on its own, to make you a competent professional email admin.

domenic commented 6 years ago

Oh, wow, thanks for looking into this further. Can you explain a bit more about the docker-mailman not working? It seems to have the database inside the container.

foolip commented 6 years ago

I haven't tried setting it up the Docker image. From my reading of https://hub.docker.com/r/alexbarcelo/mailman3/ last time, given the POSTGRES_HOST configuration I surmised that the database would be external, but maybe it'll use an internal one if not.

foolip commented 6 years ago

Filed https://github.com/whatwg/misc-server/issues/75 for what remains.

domenic commented 6 years ago

Also filed https://github.com/whatwg/misc-server/issues/77