Upgrade Redash deployment to latest release

[blarghmatey]

We have been running Redash via Salt for several years at this point and the current installation is using an out of date version. The upstream Redash project recently published a v10 release that we would like to migrate to.

As part of this upgrade we also want to remove the dependency on SaltStack and use our new Pulumi/Packer/PyInfra toolchain to support it. The upstream project publishes a docker-compose file that we can use as the basis for our installation.

Some things to consider for this deployment are:

• The release that we are running is still using Celery workers but the newer versions migrated to use rq which may necessitate some changes to our configuration
• The workers are the processes that handle any query execution against the various data sources, meaning that they are going to be the most resource intensive part of the installation. We may want to consider running those separately from the web nodes.
• We currently have Redash running in our operations VPC but we would like to move this to our Data VPC for the new installation. This will likely require some updates to security groups to allow access to the various data sources being queried. The new security group should be labelled something like business-intelligence so that it can be applied seamlessly to any other BI service that we may migrate to in the future

[Ardiea]

Current State

Two new sites created with NGINX infront of a docker container running redash.

https://redash-qa.odl.mit.edu/ https://redash-production.mit.odl.edu

Everything seems to be working well? No complaints from users who have tried things so far. I believe Peter found some things that were hard-coded to 'bi.odl.mit.edu' in the dashboard but taht is okay because we will change the DNS (see below).

TODO

1. Pick a date and time to formally cut over the environment. 
2. Message users about 'Forgot Password' link to use going forward rather than touchstone.
3. Take a look at @pdpinch's ticket here https://github.com/mitodl/ol-data-platform/issues/289 to remove the 'Remote Login' button. Should be possible.
4. Do a final dump and load from https://bi.odl.mit.edu -> both new envs. Follow instructions in ol-infrastructure/src/ol-infrastructure/applications/redash/README.md on order of operations (involves invoking a manage command to upgrade v8 database to v10).
5. Delete DNS entrie for bi.odl.mit.edu in Route53
6. Redeploy https://redash-production.odl.mit.edu -> https://bi.odl.mit.edu (Should be just a pulumi stack deployment).
7. Verify site functionality @ https://bi.odl.mit.edu
8. Rejoice.

[Ferdi]

What is the reason for removing touchstone integration ?

[Ardiea]

I defer to Tobias but I believe it came down to:

"The value of the time required to get it working and functional again > value of having it working, particularly when weighed against the fact that there is very little confidence that Redash will remain the BI solution of choice going forward."

[blarghmatey]

That's a good summary of the situation. There are a few factors that play into this choice, though I'm happy to discuss further if there is a strong reason for leaving it in place.

• The current implementation of SAML integration on Redash is dependent on a custom build of Nginx that relays to a shibboleth FastCGI service on the instance, which then passes a header back to Redash for authentication. This implementation is currently tied to our Salt infrastructure. We can explore porting it into a container, but the total time required is as-yet unkown
• The SAML integration built into Redash itself is not properly documented and does not appear to properly support any "self-hosted" options (e.g. not Okta), so there is an unknown amount of time and effort required to get that operational
• Using Touchstone means that anyone with an MIT account has access to our Redash installation
• Redash as a project has an uncertain future since their acquisition by Databricks (they are already shutting down their hosted offering). This means that any substantial investment in upkeep and integration needs to be gauged critically against time that could be spent evaluating and possibly migrating to an alternative.