Issues with security and reliability of our infrastructure

AlexDaniel commented 5 years ago

See https://github.com/matrix-org/matrix.org/issues/371.

Also maybe:

Basically, there's some perl 6 infrastructure that is used to host a bunch of stuff, including rakudo tarballs and msi's. I guess it's just a matter of time before things gets hacked? There's no hardening of any sort that I'm aware of, and definitely no policies to make things more secure. Also, last time I looked I saw a bunch of ssh keys of people who were no longer actively involved in the project, and at least one key of someone who is no longer alive.

I think a lot can be learned from https://github.com/matrix-org/matrix.org/issues/371.

Also, I don't think that fixing a few things will cut it. IMO we need to be taking steps with much broader scope when it comes to security.

AlexDaniel commented 5 years ago

meta label because nothing else fits. If someone wants to start a new label, please let me know.

lizmat commented 5 years ago

For some reason I cannot see what labels are possible, nor can I add a label or assign it. If this is intentional, that's fine by me. If not, please give me access :-)

I would think a "infrastructure" label would be appropriate here. Using "meta" here just feels like "dunno" here.

AlexDaniel commented 5 years ago

@lizmat the list of labels is here and creating new ones is possible as long as somebody is ready to be assigned for that area of expertise. Who'd be that person for infrastructure?

AlexDaniel commented 5 years ago

Ah, as for not seeing all of the labels, this wasn't exactly intended… But it's also not wrong, as in people shouldn't be adding more labels without modifying the README.

Altai-man commented 5 years ago

With the recent outage of our main server, it is obvious we have to take measures to not only secure "all our bases", but to make them reliable and keep them reachable in case of a failure.

Unfortunately, I lack necessary admin skills for planning out things. Probably ping @kawaii?

Other than this, I can provide some funding for necessary additional servers / services. Not insane amounts of money, but hosting is relatively cheap right now, and not so much resources are actually needed: current DO droplet own by me is the smallest one, and its configuration is much more than enough for the load.

As for the ticket title... I would change it to something more general like "Review, analyze and improve-secure our infrastructure". And for it to not be "too broad" for people to think, maybe add a list of things that have to be considered, including security, reliability, keeping ways to handle possible failures and so on.

rba commented 5 years ago

@lizmat the list of labels is here and creating new ones is possible as long as somebody is ready to be assigned for that area of expertise. Who'd be that person for infrastructure?

I would volunteer to take care of infrastructure stuff. Would be great to have someone else on this topic as well yet.

AlexDaniel commented 5 years ago

@rba can you review what we have now and propose some initial changes? Also, are you on IRC?

AlexDaniel commented 5 years ago

Ping @moritz, @niner, @jnthn.

kawaii commented 5 years ago

I forgot that I'd been pinged here. I'd be happy to get involved looking at the infrastructure side of things and helping to maintain that too.

niner commented 5 years ago

If we need hosting/hardware, I can add some as well. As well as the server located in Germany that's hosting camelia and has all the infrastructure needed to run additional VMs, I could also offer hosting of VMs on Atikon's company infrastructure. We're running a failover cluster behind a DDOS protected proxy.

Is there some documentation of our current website and infrastructure setup? How does it all work? And where?

AlexDaniel commented 5 years ago

There's some info on https://github.com/perl6/infrastructure-doc/

AlexDaniel commented 5 years ago

Also, any thoughts on dockerizing all the things we have? Will it help?

kawaii commented 5 years ago

Yes, I can set aside some time to Dockerize our websites and docs if everyone else agrees with that course of action.

AlexDaniel commented 5 years ago

if everyone else agrees with that course of action

Personally I'm looking for a person who would be our subject-matter expert on that topic. We will add a new label (infrastructure), and then it'd be possible to tackle smaller issues in separate tickets. That person can do such decisions alone, though general problem-solving repo rules apply so some review will be required from others.

AlexDaniel commented 5 years ago

OK, if somebody wants to be that person, here's a new ticket for you: https://github.com/perl6/problem-solving/issues/17

Altai-man commented 5 years ago

Adding my 2cents: a proposal should be written by a brave hero that will apply and others who are familiar with topic can review it, probably in a round or two.

Questions that should be addressed are:

What resources do we have now? Domain names, servers, bots to host, ecosystem data providers, installation files and so on.
How are we handling those resources now?
Our primary goals for those resources: maintainability, reliability, security and so on.
How our infrastructure should be organized to match resources with goals? E.g. "We want two containers behind a load balancer with a heartbeat indicator, because it will be Foo and Bar, also Baz and we'll love it".
How can we migrate our current infrastructure? The plan here will likely be a bit abstract, without details like "Three certain people, X, Y and Z will have keys, also the server IP will be \d.\d.\d.\d", but more of something like "Our docs site has to be containerized, deployed, we want to store a configuration in perl6-infrastructure organization repository....". This can be separated into sub-tickets and implemented over time.

rba commented 5 years ago

Haven't been on IRC very often before. Yet I have now setup a bouncer and try to handle it better this way. My nick is 'rba'.

Would recommend we start with DNS, to be sure control over perl6.org and it's subdomains are under control of the community, rather in the hand of a single person.

I've used the following services so far:

https://www.metanet.ch/email-domains/dns-hosting (5 domains are free; Website in German)
http://freedns.afraid.org (free or for 5 bucks a month)

I agree on dockerizing the services, as this would give us the flexibility. Yet I'm not an expert in this area yet, as I do small steps with kubernetes ecosystem atm.

And I agree with starting with making a roadmap and to do it's best to have kind of inventory. Subdomains, services, etc. asl Altai-man mentioned already.

jnthn commented 5 years ago

I'm glad to see there's folks willing to work on infrastructure stuff. My few thoughts, since I was pinged:

Eliminating situations where only one person can do X is good. The big blocker in the recent outage was getting DNS changes done. Using a service like that suggested by @rba sounds ideal, then we just need to nominate some folks who have access to that.
I hold rakudo.org and moarvm.org. Ideally I'd point those domains at the same name service used for perl6.org, again to avoid a single point of failure there should I be the person who can't respond quickly enough in the future.
I'm in favor of an infrastructure label here, and would encourage e.g. @rba to propose themselves on #17 as the subject matter expert.
I'm generally in favor of containerization, to ease deployment and try to reduce the amount of knowledge needed to get things running. It's not a silver bullet, but in terms of people being able to bring up and host replacement or additional/mirror instances of sites, "just build this container and run it and expose it" is quite undemanding.

maettu commented 5 years ago

/me also offers to volunteer.

AlexDaniel commented 5 years ago

@rba, @maettu please leave a separate comment on https://github.com/perl6/problem-solving/issues/17

AlexDaniel commented 5 years ago

Now that there's a dedicated label, handing this over to @rba and @maettu.

JJ commented 5 years ago

El mié., 24 abr. 2019 a las 15:31, Aleks-Daniel Jakimenko-Aleksejev (< notifications@github.com>) escribió:

Also, any thoughts on dockerizing all the things we have? Will it help?

Docs are doc-cerized (pun intended), and also mirrored now in github pages. It might help with deployment, because right now it's a bit like you need to know the repo as well as a few incantations to get the infrastructure deployed with automatic update...

rba commented 5 years ago

Quick update:

I'm on the way to sort my thoughts and write a "perl6-infra proposal", which will cover some, bot surely not all aspects. The idea is, even though I know, that I haven't got a complete overview, to write down the current state and the options Matthias and I see.

As I have the chance to talk to @lizmat and @maettu this weekend, I hope to have something next week.

I will then use github and the PR feature to add the proposal document so everyone has the chance to comment on the infrastructure ideas.

AlexDaniel commented 5 years ago

@rba any news?

moritz commented 5 years ago

I can report that the p6c.org DNS zone is now in a shared cloudflare account, so that the community administrators have access to change the records.

The same process is under way for perl6.org

jnthn commented 5 years ago

rakudo.org and moarvm.org have also had their nameservers changed to point at the shared cloudflare account.

moritz commented 5 years ago

perl6.org has also been transferred:

$ whois perl6.org|grep Name\ Server
Name Server: VAL.NS.CLOUDFLARE.COM
Name Server: CLINT.NS.CLOUDFLARE.COM

Raku / problem-solving

Issues with security and reliability of our infrastructure #9