search

alan-turing-institute / data-safe-haven-team

Project board for the Data Safe Havens in the Cloud team
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

Deploy TREEHOOSE #2

Closed jemrobinson closed 1 year ago

jemrobinson commented 2 years ago

We would like to evaluate TREEHOOSE and compare its features to the Data Safe Haven.

What are we looking for?

See #5 for details

Cost

see #6

jemrobinson commented 2 years ago

2022-10-26 Discussion with Simon and Ed

edwardchalstrey1 commented 2 years ago

Setting up AWS at Turing

Steps that were taken by the Trustworthy ID project:

  1. price it up with https://calculator.aws/#/
  2. get PMU to authorise the budget
  3. then get IT to set up the account - which in our case we'll probably do ourselves instead of relying on IT
jemrobinson commented 2 years ago

Presumably someone needs to add a credit card or set up invoicing? Can you check with the Gates PMU contact how that was done? For us I think @Arielle-Bennett is the right person to talk to.

edwardchalstrey1 commented 2 years ago

Presumably someone needs to add a credit card or set up invoicing? Can you check with the Gates PMU contact how that was done? For us I think @Arielle-Bennett is the right person to talk to.

Ok, I'm going to send an email now - in the meantime while we wait for a response, I guess we can at some point look at the AWS calculator and plug in the anticipated components needed for Treehoose #6

manics commented 1 year ago

I've opened a TREEHOOSE PR to update the version of the underlying Service Workbench, along with some minor docs improvements https://github.com/HicResearch/TREEHOOSE/pull/89 I've got another repo with a GitHub deployment workflow which I can anonymise and open-source if it's useful.

edwardchalstrey1 commented 1 year ago

Hi @manics it would be handy if you could comment on #6 - we're looking to get a rough estimate before we request budget for our TREEHOOSE deployment

edwardchalstrey1 commented 1 year ago

@jemrobinson to keep to appraised as to progress: At the moment, RCP are blocked by having to deal with questions from AWS support - @fedenanni suggested it could be ready before the start of December, but even that we aren't 100% sure

fedenanni commented 1 year ago

@jemrobinson @edwardchalstrey1 I'd keep my expectation low on "the start of December" because AWS seems quite slow in processing our change in the way payments are handled (from credit card to invoice), which is a prerequisite before setting up your account. If there's any progress during the week I'll let you know, otherwise I think it would be safe to just postpone everything to January as it seems that most of us will be on leave during December.

edwardchalstrey1 commented 1 year ago

@fedenanni are RCP still blocked with progress on the Turing AWS "account" (or organisation, whatever the correct term is)?

edwardchalstrey1 commented 1 year ago

Hi @manics I'm yet to attempt a deployment of TREEHOOSE for the purpose of comparing and contrasting with Turing-DSH and Azure TRE, but in the meantime I wondered if there was any user documentation for researchers working in a TREEHOOSE TRE, or alternatively some part of the developer/operations docs that explains what a researcher would see when logged into a desktop of a TRE instance, which apps are available etc.

The github docs mostly seem to pertain to the AWS infrastructure and operations etc as oppose to the features of TREEHOOSE. For example in DSH we have a list of available software, and the user guide shows researchers how to use things like GitLab, CodiMD alongside the usual account creation and setup steps

manics commented 1 year ago

There's a few screenshots in https://www.manicstreetpreacher.co.uk/hic-presentations-public/20220329-ukri-cloud-workshop-talk/ We're working on a proper website for our user-facing docs, but there's some info in https://hic-docs.atlassian.net/wiki/spaces/HKB/pages/870580396/How-to+articles

The available applications are down to the TRE administrator- we build all our machine images with Packer but by default there are no applications in the workspaces- open-sourcing the repo with our Packer templates is on our todo list.

edwardchalstrey1 commented 1 year ago

Thanks Simon!

edwardchalstrey1 commented 1 year ago

Deployment logs

Issue 1

Step 1A of prerequisites I am getting a permissions issue error when attempting to upload the yaml:

Screenshot 2023-01-10 at 11 29 10

Proposed solution to issue 1

I think I need to add this permission to the AWS account somehow, but where?

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:*"
            ],
            "Resource": "*"
        }
    ]
}

Screenshot 2023-01-10 at 11 46 24

manics commented 1 year ago

Skip all the landing-zone stuff, that needs to be done in the top-level organisation/management account. The idea is your organisation admin can set policies that apply to all AWS accounts in the organisation. Since you've only got a single account you won't have access to it.

manics commented 1 year ago

Having said that.... this will block other cloudformation templates. Since you're using an SSO account you'll assume a role chosen by the org admin (you might have a choice of roles), can you ask them for all the permissions they've granted to you?

edwardchalstrey1 commented 1 year ago

Turing IT have advised that I am in a group that has SystemAdministrator privileges, and that "We have top-level polices that deny creating IAM accounts, deleting CloudWatch logs and leaving the organization." Makes sense because when I log into management console I did so as SystemAdministrator: Screenshot 2023-01-10 at 14 03 15

So I don't know @manics if you think those policies will prevent me deploying TREEHOOSE completely or which part of the docs I could skip ahead to?

manics commented 1 year ago

https://github.com/z0ph/MAMIP/blob/master/policies/SystemAdministrator CloudFormation is notable for it's absence!

https://github.com/HicResearch/TREEHOOSE/blob/7d4faa178bda968dc8c818a636ae0eea89f6b5f3/src/deployment/DeploymentInstance-Cfn.yaml#L262-L541 from https://github.com/HicResearch/TREEHOOSE/pull/89/

contains the latest minimal set of permissions I could find, though it might still assume the presence of some very basic setup (like the ability to run cloudformation in the first place).

edwardchalstrey1 commented 1 year ago

Ok, sounds like that could be problematic then! I will discuss internally how feasible getting higher privileges is

edwardchalstrey1 commented 1 year ago

After chatting with @manics we concluded the most important thing in the pre-requisites was AppStream, however looks like even this will require the cloudformation:DescribeStacks action

edwardchalstrey1 commented 1 year ago

Hi @manics unfortunately we're still struggling to get an AWS setup with the correct privileges to deploy Treehoose here at Turing - I asked this before but unfortunately I don't remember your answer - as an alternative, or at least in the meantime, is there a Sandbox Treehoose environment you have already deployed that I could be given user access to?

manics commented 1 year ago

@edwardchalstrey1 Sure! Are you OK with waiting a few days so I can update the deployment (it's a bit behind at the moment)?

edwardchalstrey1 commented 1 year ago

@edwardchalstrey1 Sure! Are you OK with waiting a few days so I can update the deployment (it's a bit behind at the moment)?

Absolutely, no rush on this, any help at all is a bonus!

edwardchalstrey1 commented 1 year ago

Closing this since this is not something we're currently still planning to do - instead, Simon has provided me with a login to https://treehoose.dev.hic.dundee.ac.uk/ so we can use this for comparison work