Bulletproof server-side & Own The Node contests

[Aaron-E-Walsh]

Hi Chris. Following on our conversation over coffee at Voltage cafe I think it's time to begin implementing the server-side with military level security from the start.

Dan, Issac and Scott I would welcome your suggestions, thoughts and participation as we start the server-side now as well.

As soon as the basic Web server is in place we can then run an "Own the Node" hacking contest similar to what we have done in the past (where a successful hack into the system actually wins that server as the prize). Here's what we've done in the past: http://TheEducationGrid.org/own_the_node.html

To start I think we should design and implement the Web server with at least the following requirements:

1) Military (or bank) level security at all levels of the system 2) Host only on an open source operating systems, such as secure Linux distro 3) Use only open source software, such as Apache, MySQL, Hadoop, etc. 4) Design for global scalability from the beginning (hundreds of millions of users) 5) Cold-storage (offline) for the majority of coins, with only a small percentage online

Requirement #4 implies a fully distributed server architecture, one in which nodes (servers) are both clustered and also geographically dispersed around the world (i.e., grid/cloud). Our iED chapters (USA, Europe, Asia, Australia, Brazil, and so forth) are the first locations to host the nodes but for the time being (as we start) we can do the development locally on a local laptop and use the USA iED server farm to host the first round of "Own the Node" contests.

Requirement #5 will be one of the last aspects to implement, as this is the step where coins are moved from a live server to an offline wallet (which is only online for a short period of time to transfer coins). We can use a secure Bitcoin wallet such as Armory, or the standard (official) Bitcoin-QT wallet/node software, both of which are open source. Or we might choose a different wallet for offline (cold) storage. In any case we'll use secure network connections (such as https) between the distributed server nodes, and also when interacting with offline/cold storage nodes.

As always, we'll start simply (KISS) and iterate as each level of the system has been tested and hardened. We can start today, for example, with merely the HTTP server (such as Apache) and an open source database to store user accounts and coins. The trick, then, is making the initial decisions on the HTTP server and database to use in consideration of our requirements: #1 through #4 are our guiding requirements out of the gate.

With this in mind what are your thoughts on the Web server, database(s) and host operating system?

Aaron

[iedchrisperkins]

Hi Aaron:

That's a tall order, and I'll need to do some research. But in general for those requirements I'd suggest:

OS: Ubuntu LTS with a proactive fail2ban and iptables firewall. Any Linux OS would do, I'm just more comfortable with Debian derivatives.

Web: nginx instead of Apache

DB: PostgreSQL instead of MySQL (sorry Oracle) and Cassandra for caching.

This type of architecture should allow us to have the kind of scalability you describe. I don't think the sort of applications your envisioning would benefit from NoSQL type databases or need Hadoop.

I'm not sure the military is the role model for IT security :) But if we were to take banks as our model, security is as much a process as it is the choice of of technologies. It would be important for us to develop a set of procedures from the beginning and implement regular audits and testing.

I think a good starting point would be to set up the initial server using the above technologies, documenting the process. Then, test the configuration (Nessus scan, Bastille) and incorporate any required changes in to the install config document. That can then be the basis for an OwnTheNode contest, where we'd take the findings and update the config document, continuously improving security.

I'll start by setting up a VM with the aforementioned configuration, documenting as I go. We can then start testing and hardening from there.

[iedchrisperkins]

Hi:

I've set up a VM to use as the basis for the configuration. I still have to clean up the documentation and fix some certs, but you can access nginx running in a chroot environment on a locked down server at

http://strongbad.ath.cx/

Please confirm that you can access it. I'll post the configuration docs in the next day or so when I clean them up and fix the remaining issues.

[dangjavageek]

@iedchrisperkins : confirmed I can see http://strongbad.ath.cx/ on port 80 => telnet strongbad.ath.cx 80 Trying 76.24.29.157... Connected to strongbad.ath.cx. Escape character is '^]'. helo

400 Bad Request

---

nginx/1.1.19

Connection closed by foreign host.

[Aaron-E-Walsh]

Hi Chris. For me it actually seemed to work. Here's what I saw upon visiting that link:

Welcome to nginx! Helpful?

Aaron

iED Summits: http://summit.ImmersiveEducation.org Immersive Education Initiative: http://ImmersiveEducation.org Personal (bio & contact): http://ImmersiveEducation.org/people/aew

On Sat, Dec 21, 2013 at 4:52 PM, dangjavageek notifications@github.comwrote:

@iedchrisperkins https://github.com/iedchrisperkins : confirmed I can see http://strongbad.ath.cx/ on port 80 => telnet strongbad.ath.cx 80 Trying 76.24.29.157... Connected to strongbad.ath.cx. Escape character is '^]'. helo

400 Bad Request

400 Bad Request

nginx/1.1.19

Connection closed by foreign host.

— Reply to this email directly or view it on GitHubhttps://github.com/GravityCoins/GravityCraft/issues/4#issuecomment-31073114 .

[dangjavageek]

On 12/21/13, 9:01 AM, Aaron-E-Walsh wrote:

Hi Chris. Following on our conversation over coffee at Voltage cafe I think it's time to begin implementing the server-side with military level security from the start.

Dan, Issac and Scott I would welcome your suggestions, thoughts and participation as we start the server-side now as well.

As soon as the basic Web server is in place we can then run an "Own the Node" hacking contest similar to what we have done in the past (where a successful hack into the system actually wins that server as the prize). Here's what we've done in the past: http://TheEducationGrid.org/own_the_node.html

I will mention that I have a hosted VM to prop up a minecraft server on and the way we access it is key based ssh logins rather than passwords. Fairly common practices, and although it's not bullet proof (you restrict access to the resources to administration only from your known machines, so the solution is only as secure as the machines you restrict the access to) although it does avoid the onslaught of incoming attacks from the internet as opposed to a password based solution open to all nodes. Most of the server forums I've been on tend to favor this approach. (https://help.ubuntu.com/community/SSH/OpenSSH/Keys)

For the server mentioned, this seems reasonable:

DNS record for 76.24.29.157: c-76-24-29-157.hsd1.ma.comcast.net PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 6881/tcp closed bittorrent-tracker 6901/tcp closed jetstream 6969/tcp closed acmsoda

I run webmin on our server as well, and that shows up as:

10000/tcp open snet-sensor-mgmt

To start I think we should design and implement the Web server with at least the following requirements:

1) Military (or bank) level security at all levels of the system

Back in the day, it used to be as simple as bringing up solaris, running the solaris security toolkit (JASS) and you could seal things off down to, well, unusable levels. 8^)

I've been in a couple of RFI/RFQ's around trusted Solaris that asked for mil spec cert and common criteria EAL4+, and at some point it gets to be pretty silly to go to higher levels of cert if the machine is to be useful outside of sitting logged on to the console, i.e., the only safe machine is the machine buried in the bunker in the book Cryptonomicon after an atomic bomb is detonated directly above it. It doesn't work anymore, but man, is it ever secure. 8^) But apart from that, I understand that it needs to withstand attack.

2) Host only on an open source operating systems, such as secure Linux distro

I haven't done a survey of options, any suggestions?

3) Use only open source software, such as Apache, MySQL, Hadoop, etc.

Agreed. One of the security papers I was looking through from USENIX 2013 has slides on cracking dropbox. Quite fascinating / illuminating in terms of techniques to get at login information and server content:

http://www.openwall.com/presentations/WOOT13-Security-Analysis-of-Dropbox/

• this is the kind of thing that would ultimately be levied at any such crypto coin service as well. Note that dropbox gets good marks for the initial obfuscation and encryption modifications they've made to python.

4) Design for global scalability from the beginning (hundreds of millions of users)

Understood. That equates to software distributed on lots of machines, potentially huge data centers, if there's a crypto calculation challenge involved then potentially a p2p network for distribution would make sense, or perhaps something like SETI / fold.it / electric sheep screen savers or the equivalent.

5) Cold-storage (offline) for the majority of coins, with only a small percentage online

Interesting. Had not thought it through this far.

Requirement #4 https://github.com/GravityCoins/GravityCraft/issues/4 implies a fully distributed server architecture, one in which nodes (servers) are both clustered and also geographically dispersed around the world (i.e., grid/cloud). Our iED chapters (USA, Europe, Asia, Australia, Brazil, and so forth) are the first locations to host the nodes but for the time being (as we start) we can do the development locally on a local laptop and use the USA iED server farm to host the first round of "Own the Node" contests.

Requirement #5 will be one of the last aspects to implement, as this is the step where coins are moved from a live server to an offline wallet (which is only online for a short period of time to transfer coins). We can use a secure Bitcoin wallet such as Armory, or the standard (official) Bitcoin-QT wallet/node software, both of which are open source. Or we might choose a different wallet for offline (cold) storage. In any case we'll use secure network connections (such as https) between the distributed server nodes, and also when interacting with offline/cold storage nodes.

As always, we'll start simply (KISS) and iterate as each level of the system has been tested and hardened. We can start today, for example, with merely the HTTP server (such as Apache) and an open source database to store user accounts and coins. The trick, then, is making the initial decisions on the HTTP server and database to use in consideration of our requirements: #1 https://github.com/GravityCoins/GravityCraft/issues/1 through #4 https://github.com/GravityCoins/GravityCraft/issues/4 are our guiding requirements out of the gate.

I saw the list that Chris put together. Good job, Chris! I haven't worked directly with most of those, and again, I haven't done a recent survey of secured OS distributions. All good ideas, though.

Best regards,

-- DanG

With this in mind what are your thoughts on the Web server, database(s) and host operating system?

Aaron

— Reply to this email directly or view it on GitHub https://github.com/GravityCoins/GravityCraft/issues/4.

[SamMakesThings]

Hello all, For me, the link displayed the same as it did for Aaron. I'm glad we're getting the backend going.

Also, I've begun working on the visual design and the Minecraft mod. I'll post to GitHub when I have anything useful.

One question: Do most of you prefer to use Netbeans or Eclipse? I've downloaded both but have little preference between them.

Thanks, -Samuel

[Aaron-E-Walsh]

Hi Scott. Glad to see your account is working just fine. I'm also looking forward to seeing your push to the repository.

Personally I don't have a preference either way, but I'm using Eclipse so that we have both it and NetBeans from the jump.

I don't think we have your brother's github account yet, but I may have missed it?

Aaron On Dec 21, 2013 8:09 PM, "sStowers" notifications@github.com wrote:

Hello all, For me, the link displayed the same as it did for Aaron. I'm glad we're getting the backend going.

Also, I've begun working on the visual design and the Minecraft mod. I'll post to GitHub when I have anything useful.

One question: Do most of you prefer to use Netbeans or Eclipse? I've downloaded both but have little preference between them.

Thanks, -Samuel

— Reply to this email directly or view it on GitHubhttps://github.com/GravityCoins/GravityCraft/issues/4#issuecomment-31076423 .

[Aaron-E-Walsh]

Thanks, Chris. I agree entirely that we start the basics, ensure they are as secure and stable as possible, and then add additional layers. Add each major step we will have an Own The Node contest, ensuring that we are not moving too fast but instead are rock solid from the foundation up.

One item I did not include requirements for is the potential for virtual machines. It may be easier & preferable to deploy our server-side configuration via virtual machines, rather than on hard metal.

My assumption is that we can move to a virtual machine at any point, but we can first get the base configuration set in order to do so. But you might want to start with a virtual machine in mind if you think it's an issue to migrate into one later?

Aaron On Dec 21, 2013 1:39 PM, "Chris Perkins" notifications@github.com wrote:

Hi Aaron:

That's a tall order, and I'll need to do some research. But in general for those requirements I'd suggest:

OS: Ubuntu LTS with a proactive fail2ban and iptables firewall. Any Linux OS would do, I'm just more comfortable with Debian derivatives.

Web: nginx instead of Apache

DB: PostgreSQL instead of MySQL (sorry Oracle) and Cassandra for caching.

This type of architecture should allow us to have the kind of scalability you describe. I don't think the sort of applications your envisioning would benefit from NoSQL type databases or need Hadoop.

I'm not sure the military is the role model for IT security :) But if we were to take banks as our model, security is as much a process as it is the choice of of technologies. It would be important for us to develop a set of procedures from the beginning and implement regular audits and testing.

I think a good starting point would be to set up the initial server using the above technologies, documenting the process. Then, test the configuration (Nessus scan, Bastille) and incorporate any required changes in to the install config document. That can then be the basis for an OwnTheNode contest, where we'd take the findings and update the config document, continuously improving security.

— Reply to this email directly or view it on GitHubhttps://github.com/GravityCoins/GravityCraft/issues/4#issuecomment-31069193 .

[dangjavageek]

Samuel (Aaron, the names are Sam Stowers and David Stowers, we'll try to get David to sign up for github...),

Either Netbeans or Eclipse should be fine, or perhaps grab both. Case in point, if you're looking at a mod using Forge Mode Loader (FML) then it already comes with an eclipse project built out. Since Sun Microystems (my employer before being acquired by Oracle) bought NetBeans and the "anti-Sun" or "Eclipse" response from IBM resulted in a pseudo-IDE war between Sun and IBM, you'll find they do a lot of the same stuff, just slightly differently. More developers tend to use Eclipse and a lot of the Java team at Oracle use NetBeans. Both are free and have support to check in / check out from git hub and cover multiple languages.

Best regards,

-- DanG

[iedchrisperkins]

Hi:

key based ssh logins

Good call Dan, that is the way we'll implement here.

want to start with a virtual machine in mind

Aaron, that's the right idea as well, I have started with that in mind. I'm using Virtual Box (Oracle via Sun) to create the VMs, it should allow us to export them to other VM formats (MS, VM Ware).

My goal is to iterate the base installation tonight(I have a few changes that I want to make), test VM transfer/cloning and publish a spec/diagram here in our GitHub tonight and have it available for testing.

[Aaron-E-Walsh]

Excellent; thanks, Chris. I'll give it a spin when you're ready!

Aaron

iED Summits: http://summit.ImmersiveEducation.org Immersive Education Initiative: http://ImmersiveEducation.org Personal (bio & contact): http://ImmersiveEducation.org/people/aew

On Mon, Dec 23, 2013 at 12:25 PM, Chris Perkins notifications@github.comwrote:

Hi:

key based ssh logins

Good call Dan, that is the way we'll implement here.

want to start with a virtual machine in mind

Aaron, that's the right idea as well, I have started with that in mind. I'm using Virtual Box (Oracle via Sun) to create the VMs, it should allow us to export them to other VM formats (MS, VM Ware).

My goal is to iterate the base installation tonight(I have a few changes that I want to make), test VM transfer/cloning and publish a spec/diagram here in our GitHub tonight and have it available for testing.

— Reply to this email directly or view it on GitHubhttps://github.com/GravityCoins/GravityCraft/issues/4#issuecomment-31131752 .

[Aaron-E-Walsh]

Thanks, Dan. I didn't catch that.

Samuel, I replied to your email in haste after replying a colleague named Scott in an entirely different conversations and so mixed up your names.

In terms of IDE "coverage" both Chris & I are using Eclipse, and Dan & Isaac are using Netbeans, so if you and David want to use either (or both) we'll be in fine shape as far as balance goes. As Dan suggested I would say that you should feel free to use either one, or both, as you see fit.

Cheers, Aaron

iED Summits: http://summit.ImmersiveEducation.org Immersive Education Initiative: http://ImmersiveEducation.org Personal (bio & contact): http://ImmersiveEducation.org/people/aew

On Sun, Dec 22, 2013 at 9:18 PM, dangjavageek notifications@github.comwrote:

Samuel (Aaron, the names are Sam Stowers and David Stowers, we'll try to get David to sign up for github...),

Either Netbeans or Eclipse should be fine, or perhaps grab both. Case in point, if you're looking at a mod using Forge Mode Loader (FML) then it already comes with an eclipse project built out. Since Sun Microystems (my employer before being acquired by Oracle) bought NetBeans and the "anti-Sun" or "Eclipse" response from IBM resulted in a pseudo-IDE war between Sun and IBM, you'll find they do a lot of the same stuff, just slightly differently. More developers tend to use Eclipse and a lot of the Java team at Oracle use NetBeans. Both are free and have support to check in / check out from git hub and cover multiple languages.

Best regards,

-- DanG

— Reply to this email directly or view it on GitHubhttps://github.com/GravityCoins/GravityCraft/issues/4#issuecomment-31101699 .

[dangjavageek]

On 12/23/13, 11:25 AM, Chris Perkins wrote:

Hi:

key based ssh logins

Good call Dan, that is the way we'll implement here.

want to start with a virtual machine in mind

Aaron, that's the right idea as well, I have started with that in mind. I'm using Virtual Box (Oracle via Sun) to create the VMs, it should allow us to export them to other VM formats (MS, VM Ware).

My goal is to iterate the base installation tonight(I have a few changes that I want to make), test VM transfer/cloning and publish a spec/diagram here in our GitHub tonight and have it available for testing.

Thanks, Chris. Was there supposed to be any ports open besides http and ssh? Did you want us to test ssh and/or did you need public key entries to add to the server so that we could try ssh login? Let us know what we can do to help.

Best regards,

-- DanG

— Reply to this email directly or view it on GitHub https://github.com/GravityCoins/GravityCraft/issues/4#issuecomment-31131752.

[iedchrisperkins]

Was there supposed to be any ports open besides http and ssh?

Nope; in fact once we get going I'd like to restrict ssh to known hosts

Did you want us to test ssh and/or did you need public key entries to add to the server so that we could try ssh login?

Yes, if you want to email me your public key we'll set it up

I spent the majority of today verifying and documenting the server configuration on a new VM I set up. I'll check this document in to our repository; this and other configuration documents should be versioned to keep up with current configurations. The document contains no secret information or passwords. There are 2 remaining to-dos on the server side: figure out the firewall rules for the chrooted host and get SSL working for nginx.

[iedchrisperkins]

Hi: I resolved one of the to-dos, the chroot firewall for the nginx user. I also updated the documentation to reflect the necessary changes. Now we just need to get SSL going for https...I'll need to generate some certs. But otherwise looks like we're almost there.

[Aaron-E-Walsh]

That's great news, Chris; in order to get SSL going do you need to purchase a cert, or can you generate one? Aaron

iED Summits: http://summit.ImmersiveEducation.org Immersive Education Initiative: http://ImmersiveEducation.org Personal (bio & contact): http://ImmersiveEducation.org/people/aew

On Tue, Dec 24, 2013 at 12:31 PM, Chris Perkins notifications@github.comwrote:

Hi: I resolved one of the to-dos, the chroot firewall for the nginx user. I also updated the documentation to reflect the necessary changes. Now we just need to get SSL going for https...I'll need to generate some certs. But otherwise looks like we're almost there.

— Reply to this email directly or view it on GitHubhttps://github.com/GravityCoins/GravityCraft/issues/4#issuecomment-31179839 .

[iedchrisperkins]

in order to get SSL going do you need to purchase a cert, or can you generate one?

Hi Aaron: For this test system, I've generated a self-signed cert and have enabled HTTPS for this server. Your browser will warn you that the cert isn't trusted, but you can continue to the site https://strongbad.ath.cx. In fact I've set it up such that regular HTTP requests are redirected to HTTPS, so all content should be served via SSL.

I've updated the configuration and versioned the config document as well.

[dangjavageek]

On 12/26/13, 2:51 PM, Chris Perkins wrote:

in order to get SSL going do you need to purchase a cert, or can
you generate one?

Hi Aaron: For this test system, I've generated a self-signed cert and have enabled HTTPS for this server. Your browser will warn you that the cert isn't trusted, but you can continue to the site https://strongbad.ath.cx. In fact I've set it up such that regular HTTP requests are redirected to HTTPS, so all content should be served via SSL.

I've updated the configuration and versioned the config document as well.

Thanks, Chris. Looks good:

76.24.29.157: c-76-24-29-157.hsd1.ma.comcast.net

port 22/tcp open ssh -> seems to work:

% ssh dang@strongbad.ath.cx The authenticity of host 'strongbad.ath.cx (76.24.29.157)' can't be established. RSA key fingerprint is 00:d8:15:14:89:8a:7f:4f:d6:e7:f0:b2:a2:3c:d3:9f. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'strongbad.ath.cx,76.24.29.157' (RSA) to the list of known hosts. _ | | _ ___ _| | _ / ` | / | | | |/ ` | | (| |/ /| || | | (| | ,/|,||,_|

Welcome to azula. All activity will be logged.

dang@strongbad.ath.cx's password:

port 80/tcp open http -> redirects to https://strongbad.ath.cx

port 443/tcp open https -> GravityCraft banner page, looks good, thanks

port 2222/tcp open EtherNet/IP-1, not sure what that is -> http://en.wikipedia.org/wiki/EtherNet/IP

Best regards,

-- DanG

[iedchrisperkins]

Hi Dan:

port 2222/tcp open

Good catch, I should have mentioned in this thread that I already have a server listening on port 22; the chrooted environment is listening on port 2222 and this is the environment that I'll grant you and the rest of the team access to via public key authentication.