[ADD] clouder_template_red_october: File encyption/decryption

[lasley]

This is a WIP, I have a question that I'll ask inline. ReadMe below for info on this template.

Clouder Template - Red October

This module provides a Clouder Template for Red Octover.

Red October is a cryptographically-secure implementation of the two-person rule to protect sensitive data. From a technical perspective, Red October is a software-based encryption and decryption server. The server can be used to encrypt a payload in such a way that no one individual can decrypt it. The encryption of the payload is cryptographically tied to the credentials of the authorized users.

Authorized persons can delegate their credentials to the server for a period of time. The server can decrypt any previously-encrypted payloads as long as the appropriate number of people have delegated their credentials to the server.

This architecture allows Red October to act as a convenient decryption service. Other systems, including CloudFlare’s build system, can use it for decryption and users can delegate their credentials to the server via a simple web interface. All communication with Red October is encrypted with TLS, ensuring that passwords are not sent in the clear.

Read More on CloudFlare's Blog

Browse Red October on Github

[codecov-io]

Current coverage is 31.54% (diff: 44.44%)

Merging #187 into master will increase coverage by 0.53%

@@             master       #187   diff @@
==========================================
  Files            73         74     +1   
  Lines          5653       5732    +79   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           1753       1808    +55   
- Misses         3900       3924    +24   
  Partials          0          0          

Powered by Codecov. Last update 8695902...d07eabd

[YannickB]

Just to be sure, redoctober is about file/data secure storage am I right ? If yes, we had a discussion about https://www.vaultproject.io/ which looked very good to me.

Can you describe what is the exact purpose for redoctober ? Shall it replace https://www.vaultproject.io/ and if yes can you explain why we shall use redoctober instead ?

[lasley]

Don't you think this template shall be merged with the cfssl template ? With your proposition we'll have

Red October and CFSSL serve different purposes. Red October is file encryption, and CFSSL is a certificate authority.

CFSSL is required to secure our internal communications, such as Logstash=>Elastic.

Red October is going to handle encryption/decryption of private keys generated by CFSSL, Openssl, OpenSSH, etc.

OpenSSL container is required in order to get some certificate information that I was otherwise unable to obtain from the other services.

As I understand it, Docker is built upon the premise of modular design & the combining of logical services is not the way to go. Technically we could just install OpenSSL on every container that requires it, but I feel like that will just increase the deploy time exponentially vs. the one container.

Can you describe what is the exact purpose for redoctober ? Shall it replace https://www.vaultproject.io/ and if yes can you explain why we shall use redoctober instead ?

I included some of this our email thread before I noticed the question here. Just so it's all public too, here's the breakdown from the email. Let me know if it doesn't clarify.

Vaultproject duplicates a lot of functionality that the CFSSL CA does, without actually providing a CA. It does technically offer more than Red October though, specifically the storage of the encrypted data vs simply key management.

On the flip side, Red October is more secure. It has another layer of decryption rights called delegation, which allows you to delegate decryption rights to different RO Vaults. This means that the RO Vault itself is actually portable, and able to be distributed amongst multiple Vaults. In our context, this means we can add superusers to decrypt the data & template those users across customer vaults.

TBH both are about as state of the art in terms of design, for the most part it’s just the question of feature duplication. I think a lot of Vault project’s appeal (at least for me) is their website. I still need to do a side-by-side comparison though, and I’m somewhat thinking of making a Vaultproject connector simply for the hell of it.

Edit: Just saw your edit. Too late, I explained anyways! 🚀

[YannickB]

Thanks @lasley for clarification, looks like the way to go for me !

[lasley]

I'm planning on doing a few tests with this later today & will report back on whether we can merge.

[lasley]

Alright this works!

I'm not sure if the proxy part is though. Am I supposed to do something more than just setting the name as https?

[YannickB]

So redoctober has a web interface... That's really good to know.

You'll need to add link to proxy/dns with https://github.com/clouder-community/clouder/blob/0.9.0/clouder_template_odoo/template.xml#L315, and allow base creation in the application. Then try to create a base on this service, with the url you want to access it.

[lasley]

So redoctober has a web interface... That's really good to know.

Yeah totally - I'm using the JSON API via the same port, but it comes with a rudimentary interface based on the JSON API. Makes it really freaking easy to poke around

$ docker run -d lasley/redoctober-exec -p 8080:8080
$ curl -k https://localhost:8080
<!DOCTYPE html>
<html lang="en">
<head>
    <title>Red October - Two Man Rule File Encryption &amp; Decryption</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no">

    <link rel="stylesheet" href="//netdna.bootstrapcdn.com/bootstrap/3.0.2/css/bootstrap.min.css" />
    <link rel="stylesheet" href="//netdna.bootstrapcdn.com/bootstrap/3.0.2/css/bootstrap-theme.min.css" />
    <script src="//cdnjs.cloudflare.com/ajax/libs/jquery/2.0.3/jquery.min.js"></script>
    <script src="//netdna.bootstrapcdn.com/bootstrap/3.0.2/js/bootstrap.min.js"></script>
    <style type="text/css">
        .footer{ border-top: 1px solid #ccc; margin-top: 50px; padding: 20px 0;}
    </style>
</head>

[lasley]

@YannickB Hmm ok so I don't think I've tried to create a Base manually before. I created #193 because the view was basically broken, but now that I have things entered I'm getting stuck on some dependencies.

I think I have everything figured out except for the application.default_image_id.version_ids: constraint. What image is supposed to be linked to the application? The exec?

[YannickB]

@lasley Application with children shall not have default_image_id, I'm wondering if I didn't let a wrong constraint behind. Can you provide a link to this constraint ?

[lasley]

@YannickB - https://github.com/clouder-community/clouder/blob/c6d231b43cbc7e7033c3f383cad40e7c6d6aad1d/clouder/base.py#L609

[YannickB]

Oh ok. This function is used when you didn't first created a service. This way you can directly create a base, it will create the service behind for the customer. Useful when you want to create a service each time you create a base for a customer, so he have his own service (no multi-base mode)

TBH, recently I only followed the process first create manually the service, then create the base and fill the service_id field so this function is not called. You should do the same for now until we debug it.

[lasley]

Ok I see, so there's just a missing deploy step if you create it the one direction. I already had a service, so that was a pretty easy fix.

Hmmm I must still be missing something though, now I'm getting a missing port error. I can replicate it by Reloading the proxy, which jives in the traceback:

Traceback (most recent call last):
  File "/opt/odoo/odoo/http.py", line 638, in _handle_exception
    return super(JsonRequest, self)._handle_exception(exception)
  File "/opt/odoo/odoo/http.py", line 675, in dispatch
    result = self._call_function(**self.params)
  File "/opt/odoo/odoo/http.py", line 331, in _call_function
    return checked_call(self.db, *args, **kwargs)
  File "/opt/odoo/odoo/service/model.py", line 119, in wrapper
    return f(dbname, *args, **kwargs)
  File "/opt/odoo/odoo/http.py", line 324, in checked_call
    result = self.endpoint(*a, **kw)
  File "/opt/odoo/odoo/http.py", line 933, in __call__
    return self.method(*args, **kw)
  File "/opt/odoo/odoo/http.py", line 504, in response_wrap
    response = f(*args, **kw)
  File "/opt/odoo/addons/web/controllers/main.py", line 862, in call_kw
    return self._call_kw(model, method, args, kwargs)
  File "/opt/odoo/addons/web/controllers/main.py", line 854, in _call_kw
    return call_kw(request.env[model], method, args, kwargs)
  File "/opt/odoo/odoo/api.py", line 679, in call_kw
    return call_kw_model(method, model, args, kwargs)
  File "/opt/odoo/odoo/api.py", line 664, in call_kw_model
    result = method(recs, *args, **kwargs)
  File "/media/sf_Repos/clouder/clouder/models/base.py", line 587, in create
    return super(ClouderBase, self).create(vals)
  File "/media/sf_Repos/clouder/clouder/models/model.py", line 432, in create
    res.do('create', 'deploy_frame')
  File "/media/sf_Repos/clouder/clouder/models/model.py", line 332, in do
    getattr(self, 'do_exec')(action, job_id)
  File "/media/sf_Repos/clouder/clouder/models/model.py", line 346, in do_exec
    getattr(self, action)()
  File "/media/sf_Repos/clouder/clouder/models/model.py", line 360, in deploy_frame
    self.deploy_links()
  File "/media/sf_Repos/clouder/clouder/models/model.py", line 394, in deploy_links
    link.deploy_()
  File "/media/sf_Repos/clouder/clouder/models/base_link.py", line 86, in deploy_
    'deploy_link ' + self.name.name, 'deploy_exec', where=self.base_id)
  File "/media/sf_Repos/clouder/clouder/models/model.py", line 332, in do
    getattr(self, 'do_exec')(action, job_id)
  File "/media/sf_Repos/clouder/clouder/models/model.py", line 346, in do_exec
    getattr(self, action)()
  File "/media/sf_Repos/clouder/clouder/models/base_link.py", line 93, in deploy_exec
    self.control() and self.deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_odoo/template.py", line 465, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_gitlab/template.py", line 456, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_piwik/template.py", line 130, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_shinken/template.py", line 250, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_proxy/template.py", line 192, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_mail/template.py", line 214, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_dns/common.py", line 64, in deploy_link
    self.base_id.generate_cert_exec()
  File "/media/sf_Repos/clouder/clouder_template_proxy/template.py", line 110, in generate_cert_exec
    proxy_link.deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_odoo/template.py", line 465, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_gitlab/template.py", line 456, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_piwik/template.py", line 130, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_shinken/template.py", line 250, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_proxy/template.py", line 263, in deploy_link
    port +
UnboundLocalError: local variable 'port' referenced before assignment

I think everything is having this trouble though, here's Gitlab Edit: maybe not Gitlab actually, but something during the Odoo one click that isn't generating an exception, but in the logs:

2017-01-03 23:10:40,677 31878 INFO clouder odoo.addons.clouder.models.model:   File "/media/sf_Repos/clouder/clouder/models/model.py", line 360, in deploy_frame
    self.deploy_links()
  File "/media/sf_Repos/clouder/clouder/models/model.py", line 394, in deploy_links
    link.deploy_()
  File "/media/sf_Repos/clouder/clouder/models/base_link.py", line 86, in deploy_
    'deploy_link ' + self.name.name, 'deploy_exec', where=self.base_id)
  File "/media/sf_Repos/clouder/clouder/models/model.py", line 332, in do
    getattr(self, 'do_exec')(action, job_id)
  File "/media/sf_Repos/clouder/clouder/models/model.py", line 346, in do_exec
    getattr(self, action)()
  File "/media/sf_Repos/clouder/clouder/models/base_link.py", line 93, in deploy_exec
    self.control() and self.deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_odoo/template.py", line 465, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_gitlab/template.py", line 456, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_piwik/template.py", line 130, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_shinken/template.py", line 250, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_proxy/template.py", line 192, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_mail/template.py", line 214, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_dns/common.py", line 64, in deploy_link
    self.base_id.generate_cert_exec()
  File "/media/sf_Repos/clouder/clouder_template_proxy/template.py", line 110, in generate_cert_exec
    proxy_link.deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_odoo/template.py", line 465, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_gitlab/template.py", line 456, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_piwik/template.py", line 130, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_shinken/template.py", line 250, in deploy_link
    super(ClouderBaseLink, self).deploy_link()
  File "/media/sf_Repos/clouder/clouder_template_proxy/template.py", line 263, in deploy_link
    port +

And looking back on this, I think we maybe are missing some proxy stuff in Elasticsearch. Oh well, we'll find out soon enough 😆

Oh and for good measure, relevant docker ps:

runbot@runbot:~/instance/runbot-addons$ docker ps | grep dev-
8341f2445665        dev-redoctober-exec-20170103.230802               "/go/src/github.com/c"   4 minutes ago       Up 4 minutes                                                                                                              dev-redoctober-exec
807291f1270b        dev-redoctober-data-20170103.230800               "/bin/sh -c cat"         5 minutes ago       Up 5 minutes                                                                                                              dev-redoctober-data
a42d939df7d5        dev-clouder9-all-clouder9-files-20170103.230411   "/bin/sh -c 'tail -f "   8 minutes ago       Up 8 minutes                                                                                                              dev-clouder9-all-clouder9-files
545804c18df4        dev-clouder9-all-clouder9-data-20170103.230406    "/bin/sh -c 'tail -f "   8 minutes ago       Up 8 minutes                                                                                                              dev-clouder9-all-clouder9-data
62f799a8c927        dev-clouder9-all-postgres-exec-20170103.230358    "/docker-entrypoint.s"   8 minutes ago       Restarting (127) 2 minutes ago   5432/tcp                                                                                 dev-clouder9-all-postgres-exec
8c8b82b862e2        dev-clouder9-all-postgres-data-20170103.230355    "/bin/sh -c 'tail -f "   9 minutes ago       Up 9 minutes                                                                                                              dev-clouder9-all-postgres-data
b5dd4bc80f54        dev-proxy-exec-20170103.230332                    "/bin/sh -c nginx"       9 minutes ago       Up 9 minutes                     0.0.0.0:30007->80/tcp, 0.0.0.0:30008->443/tcp                                            dev-proxy-exec
3f586fa7f9ed        dev-proxy-data-20170103.230329                    "/bin/sh -c 'tail -f "   9 minutes ago       Up 9 minutes                                                                                                              dev-proxy-data
8f44b46ea3e2        dev-postfix-data-20170103.230314                  "/bin/sh -c 'tail -f "   9 minutes ago       Up 9 minutes                                                                                                              dev-postfix-data
41c6c090d1e6        dev-bind-exec-20170103.230206                     "/bin/sh -c 'named -g"   10 minutes ago      Up About a minute                53/tcp, 0.0.0.0:30003->53/udp                                                            dev-bind-exec
33a54fbe9319        dev-bind-data-20170103.230203                     "/bin/sh -c 'tail -f "   10 minutes ago      Up 10 minutes                                                                                                             dev-bind-data
cf8161510b2e        dev-backup-bup-20170103.230140                    "/bin/sh -c 'supervis"   11 minutes ago      Up 11 minutes                    0.0.0.0:30000->5666/tcp, 0.0.0.0:30001->8080/tcp                                         dev-backup-bup

[YannickB]

Hum I guess we're not entering this condition :

https://github.com/clouder-community/clouder/blob/master/clouder_template_proxy/template.py#L253

Can you output self.base_id.service_id.ports to see where is the problem ? [IMP] : I guess we shall raise an issue here when neither http nor https are found in self.base_id.service_id.ports

[lasley]

Service with NAME: dev-redoctober, PORTS: {}

[lasley]

And this makes sense because the service doesn't seem to have any exposed ports:

[YannickB]

That's strange since you specified the port in

https://github.com/clouder-community/clouder/pull/187/files#diff-067c1eb14f6f8af45482f279b63141a4R7

But I see what the problem is. You have to expose it, either local or internet in order for the port to be inherited in service.

Eg. https://github.com/clouder-community/clouder/blob/master/clouder_template_odoo/template.xml#L122

[lasley]

Damn still no go. It looks like Let's Encrypt is failing on the proxy before this (my dev doesn't have a public port 80). Think it's something to do with that? Should I maybe try a Base for something else that's known good (and what would that be)?

[YannickB]

Hum, to be honest the local expose status wasn't much tested until now, can you try with internet exposed ?

The local shall be used only for container to container interactions, so it'll not publish port in the container itself.