Disclosure of Cloud Infrastructure Code and Documentation

[RomanNess]

What is missing

In contrast to the backend & mobile apps that are fully open-source and well documented, none of the infrastructure code was published and there is very little documentation about the cloud infrastructure. From the existing documentation we learned that the backend will run on a K8s Cluster on the OpenShift Platform and that Telekom will be responsible for operations (see cwa-server/architecture-overview.md#overview, cwa-verification-portal#contributors and cwa-verification-server/architecture-overview.md).

We would love to see you publish as much of the infrastructure code as possible as open source and enable the community to review and contribute.

Why should it be included

• The remaining code chose the open-source development model to enable full transparency and to allow peer reviews by experts and the community. Infrastructure as code (e.g. Terraform, Helm Charts, etc) allows a similar development process for infrastructure code with equal benefits.
• Similar to bugs in backend applications, misconfigurations in infrastructure as code can also lead to security risks. Security by obscurity should not be a practice to mitigate these risks. Please note, that we don't want to question the capabilities of the operations teams.
• Many backend developers are also capable to provision cloud infrastructures and can use their gained knowledge of the cwa backend services to review whether the infrastructure is suitable to run the services under the expected load. In our opinion, there should be no strict separation between Dev and Ops when developing cloud-native applications.

Suggested change

Publish as much of the infrastructure code and documentation as possible as soon as possible. We are convinced that the project will benefit greatly once the community can contribute to it. We understand if you prefer to wait until after the first release of the app. A compromise might be the disclosure of the infrastructure code in an upcoming release.

---

Internal Tracking ID: EXPOSUREAPP-2072

[SebastianWolf-SAP]

Thank you very much for this issue. Actually, we are planning something like that - it's unclear yet how much we can go into detail, but you will most probably see something. We already discussed that in the issues https://github.com/corona-warn-app/cwa-server/issues/451 and corona-warn-app/cwa-documentation#183. As your issue goes beyond the latter one, we will close corona-warn-app/cwa-documentation#183 accordingly.

We will keep this issue open until we have something ready.

Mit freundlichen Grüßen/Best regards, SW Corona Warn-App Open Source Team

[egandro]

Dear @SebastianWolf-SAP,

disclosing the the operation process of the cloud and having a written documentation as mentioned in corona-warn-app/cwa-documentation#183 are very different stories.

Please keep in Mind - the source of the infrastructure will not reflect the Operation requirements and processes needed for the operators.

corona-warn-app/cwa-documentation#183 is also about referencing the basis of security that is required for Postgres Operation. The mere existence of the code doesn't explain the "why". However the IBM Tivoli Operation Manual - given as an example - covers this in depth.

[SebastianWolf-SAP]

Dear @egandro,

the original request of this issue includes the following: "Publish as much of the infrastructure code and documentation as possible as soon as possible." As the Postgres database is part of the Cloud infrastructure, this issue covers corona-warn-app/cwa-documentation#183.

Mit freundlichen Grüßen/Best regards, SW Corona Warn-App Open Source Team

[RomanNess]

We created a list of possible items that might be included in the disclosure of infrastructure code and documentation. Please don't hesitate to comment if you can think of additional items. I will try to keep this list up-to-date.

Code

• Infrastructure as code for the cloud infrastructure (Terraform?)
  • [ ] OpenShift/AppAgile cluster
  • [ ] Autoscaling Group
  • [ ] RDS
  • [ ] VPCs + subnets
  • [ ] load balancers
  • [ ] object storage + CDN
• Helm Charts (or K8s manifests)
  • [ ] cwa-server
  • [ ] cwa-verification-server
  • [ ] cwa-testresult-server
  • [ ] cwa-verification-iam
  • [ ] cwa-verification-portal
  • [ ] values yamls for Helm Charts (minus credentials)

Documentation

Cloud

• [x] How is logging/monitoring implemented?
  • Monitoring & Loggin performed from a separate OTC tenant
  • Central graylog in AppAgile
  • T-System's CDC (Cyber Defense Center) is responsible for security monitoring
• [ ] How is the cloud infrastructure provisioned?
• [x] Which OTC services are used under the hood?
  • AppAgile / Openshift
  • OTC RDS
  • Magenta CDN

Database

• [ ] Database operational manual

  EDIT: I just saw that some documentation was already published at https://github.com/corona-warn-app/cwa-documentation/blob/master/backend-infrastructure-architecture.pdf. We will check if some of the items are already covered there.

[kreincke]

Dear @RomanNess :

We did not forget you! We hope, that some of your questions have already answered by the infrastructure documentation we have already published. Additionally, we seriously discussed your suggestions in different meetings: If we understand you correctly, we would have to publish the complete code and installation docs of our underlying cloud infrastructure if we fulfilled in the final sense what you had proposed That would blast the focus of this project and - to be honest - is not part of our internal plannings.

Nevertheless: some of the wishes you have listed make a lot of sense in the context of the corona warn app, for example publishing the respective helm charts. But we still have to discuss some side effects before we can release the information. When it will have become possible to release it we will do so! So, for the moment, we have to ask for your patience.

[RomanNess]

Thank you for the update @kreincke!

I edited the list in my previous comment to reflect information gathered from the already published infrastructure documentation. And I want to emphasize that it also answered numerous questions in my head that are not reflected in the list.

If we understand you correctly, we would have to publish the complete code and installation docs of our underlying cloud infrastructure if we fulfilled in the final sense what you had proposed.

Yes, that would be the maximum disclosure that you could offer. Though, we are aware that this is not in the interest of the operations team or might even pose security risks in itself. However, we are still convinced that the project will benefit from as much disclosure as possible. I think a more realistic goal for your team might be to publish anything that you can and to explain why it is not feasible to publish the rest. In the end, even security by obscurity might be a valid reason in some corners of the infrastructure depending on its structure.

Nevertheless: some of the wishes you have listed make a lot of sense in the context of the corona warn app, for example publishing the respective helm charts. But we still have to discuss some side effects before we can release the information.

Is it possible for you to state the possible side effects that you are discussing? Please note, that it is not my primary intention to argue against them. I'm personally curious if there are side effects that I haven't thought of.

Modern infrastructure as code tools such as Terraform offer the possibilities to separate values and secrets from the actual infrastructure code (in the documentation you also state that the project uses HashiCorp Vaults). The same does also apply for Helm Charts and values.yml files. So it should theoretically be possible to publish the structure of your infrastructure through code without disclosing any values or secrets.

[Ein-Tim]

Same here, we did not see any update on this in more than one year. Please give us an update and also let us know if this is still considered internally.