The shipment tracker should be hosted in the EU on servers owned by a European company :eu:

[coderbyheart]

The shipment tracker's production (and testing instances) should be operating using EU-based companies.

DevOps tasks

Number of concurrent users will be fairly low, so scalability is not key.

• [x] find a solution that allows to continuously deploy the shipment tracker (push to deploy)
  • [x] Clever Cloud (#300, we tried that but the project structure was getting in the way, see below) - follow up in #314 works
  • [ ] sloppy.io (looks good, after containerizing the shipment tracker, see #303)
  • [ ] hosting option by @dch #319
• [x] solution needs to be maintained via Git (AKA GitOps)
  • for Clever Cloud: #343
• [x] Create an Admin Interface, to migrate the database, including a pgMyAdmin instance to peek into the db
  • PostgreSQL is accessible from the internet, so any local running client can modify the database (including sequelize-cli)
• [x] Set up regular, encrypted backups of the database
  • Clever Cloud has automated backups for PostgreSQL:

    This add-on is automaticalty backuped, on a daily basis, with a 7 days retention. For longer retention of backup frequency, consider one of the enterprise plans.

• [x] Set up a EU-based email provider for sending emails, e.g. mailbox.org or rapidmail.com

Work needed on the shipment tracker itself

• [x] Replace auth0 with passport.js #314

Background

:information_source: This is a summary of my concerns posted in Slack, to better track this effort and solicit volunteers.

The hosting of this project should be EU-based, because we mostly store data from EU citizens. Using a US-based hoster has multiple implications. Per GDPR we are required to get their consent that their data is to be stored outside of the EU, which we don't do at the moment. Not only would we store volunteer data in the US, but would expose their meta data (like IP addresses, browser versions, etc) to a US based company which can be used to profile aid workers. Even though US companies will state the they comply to GDPR, the US intelligence services have shown over and over again that they will pressure US based companies to lower protection of customer data. In my opinion we should not rely on US companies for storing personal information of aid workers in a field of work that we know is highly political and where exposure of that data can impact their personal lives.

I recommend to look into alternatives for that. I know the founder of https://www.clever-cloud.com/ and I am sure they would provide us with free hosting. If we need Kubernets, I am also confident https://www.giantswarm.io/ would be open for sponsoring.

I am aware that I recommended auth0 in the past, which is also a U.S. based company. Back then I didn't consider the implication. Given that we are in a very early stage of the application and we basically don't use any specific features of auth0, this is still a good time to reconsider this decision. There are European based alternatives (like https://www.cidaas.com/), but we could also go with http://www.passportjs.org/ which has been around for years to provide Express.js apps with authentication (which would mean we store usernames and passwords ourselves, however).

Previous attempts to deploy to Clever Cloud

See #300

(Original post)

I tried to deploy the shipment tracker (note that I have limited experience deploying containerized solutions), and here are my thoughts on what I think needs to be improved:

1. getting the server up and running is very involved: generating code, compiling typescript. I think this should be done once and not repeated when spinning up a new instance.
2. this could be done as part of a GitHub release process: compile the code, run all tests, and publish a release, and attach the compiled code as artifact.
3. That way we also will have reproducible builds, and know which exact version of the source code we are running.
4. Because the frontend is a static web application, it should not be run through the same express instance. It can be put on a plain webserver with proper long-term caching set up. Outcome of 2 should be everything needed to get the web app running, so I can unzip the artifact on a webserver and be done with the deployment.
5. This would also allow to decouple the deployment of frontend and backend, because they do not change all the time for the same reasons.
6. Right now starting the server runs database migrations. Once we start scaling backend server instances we could hit a situation where multiple instances run the migrations. I hope they use transactions, but then some migrations might fail. Migrations on prod should only be run by one instance.

[maltekrupa]

Hey, 👋

Good work! I think you got the most pressuring things sorted already. Even though I haven't heard of sloppy before, they seem like a good solution for your container hosting needs. Well documented, not too complex and not too expensive.

The GitHub actions pipeline you described in #303 also sounds good. Keep in mind to avoid the latest tag when deploying your containers.

The database backup could be an automated snapshot at the beginning and eventually move to what you described: an encrypted backup. I'd recommend to move it somewhere outside of sloppy; encrypted with something trustworthy.

The pgadmin interface can also run in a dedicated container at sloppy. Having it (at least) behind a basic auth, so no one can brute force their way into your database, is a must. I know it is annoying, but it is worth it.

About the eMail topic: Pick whatever makes you happy or fulfils your legal requirements. For sending eMails you'll need at least an SPF record in your DNS. DKIM would also be nice. Both reduce the chance to have your mails be detected as spam. I'm not aware of any eMail provider which do not already have everything in place to guide you through this maze of acronyms. :)

Malte

[deammer]

@coderbyheart if we're hosting the backend separately from the frontend, I would suggest using Netlify to host our React app. It's very easy to integrate and volunteers are likely to be familiar with it. Netlify has servers throughout the world, including the US, but we wouldn't be storing user information on their servers so I don't think it's a problem.

What do you think?

[coderbyheart]

The problem here again is that user are accessing servers operated by a US based company. And since they hit their webserver they transfer a lot of identifying information.

I will clarify the scope of this issue.

[coderbyheart]

I've set up RapidMail for the shipment tracker. Docs have been updated as well.

[coderbyheart]

Closing as we have achieved the goal.