Service Workbench on AWS has been moved to maintenance mode. While in maintenance, we will not add new features to this solution guidance. Security engagement should be directed to AWS Security at aws-security@amazon.com. If you are new to Service Workbench on AWS, we recommend that you explore using Research and Engineering Studio on AWS (https://aws.amazon.com/hpc/res/). You can get started by following instructions in the Research and Engineering Studio User Guide (https://docs.aws.amazon.com/res/latest/ug/overview.html). If you are an existing customer of Service Workbench on AWS and have additional questions or need immediate help, please contact your AWS Account team.
Service Workbench on AWS is a cloud solution that enables IT teams to provide secure, repeatable, and federated control of access to data, tooling, and compute power that researchers need. With Service Workbench, researchers no longer have to worry about navigating cloud infrastructure. They can focus on achieving research missions and completing essential work in minutes, not months, in configured research environments.
With Service Workbench on AWS, researchers can quickly and securely stand up research environments and conduct experiments with peers from other institutions. By automating the creation of baseline research setups, simplifying data access, and providing price transparency, researchers and IT departments save time, which they can reinvest in following cloud best practices and achieving research reproducibility.
Service Workbench integrates existing AWS services, such as Amazon CloudFront, AWS Lambda, and AWS Step Functions. Service Workbench enables you to create your own custom templates and share those templates with other organizations. To provide cost transparency, Service Workbench has been integrated with AWS Cost Explorer, AWS Budgets and AWS Organizations.
There are three types of Studies available in Service Workbench: My Studies, Organizational Studies and Open Data. Once you have created a Study you can upload files to it. Organizational Studies can be shared with other members in the organization. Owners of a study can amend the permissions of the study to grant access to other users. Once you have found the study or studies in which you are interested to perform research, you can deploy a workspace to attach the data to and conduct your research.
This is the account where Service Workbench infrastructure is deployed.
This is the account where compute resources are deployed.
Service Workbench contains the following components.You can find these components under the
Infrastructure: The following AWS resources are created as part of this component deployment:
The solution also includes a Continuous Integration/Continuous Delivery feature:
Service Workbench documentation can be accessed in the PDF format or by visiting the AWS Solution Implementation Guide.
For information on installing Service Workbench on AWS, please visit the AWS Solutions Implementation Guide.
You can view the online documentation if you do not have Service Workbench locally installed on your machine. Click the following links to access the documentation:
npm install -g pnpm@latest-8
Go
is used for creating a multipart S3 downloader tool that is used in AWS Service Catalog EC2 Windows based research environments.For more information, refer to the Prerequisites section of the Service Workbench Deployment Guide.
To create the initial settings files, take a look at the example.yml settings file in main/config/settings/example.yml and create your own copy. The stage is either 'example' or your username. This method should be used only for the very first time you install this solution. In the rest of this README, \$STAGE is used to designate the stage.
For more information, refer to the Prepare Main Configuration File section of the Service Workbench Deployment Guide.
You can run the Service Workbench installation from your local machine or an Amazon Elastic Compute Cloud (Amazon EC2) instance. The installation involves the following:
Now, let's perform an initial deployment. Note that when invoked without parameters, this will assume a deployment stage of \$USER, which is the logged-in user name on Mac and Linux systems.
scripts/environment-deploy.sh
You can override the default stage name of \$USER if you prefer. For example, if you want your stage name to be qa
, then:
scripts/environment-deploy.sh qa
In case you have made some changes to the Service Workbench components after the initial deployment, use these commands to re-deploy these components individually. There won't be any change to your installation if you have not changed any of the components.
Following an initial successful deployment, you can subsequently deploy updates to the infrastructure, backend, and post-deployment components as follows:
cd main/solution/<component>
pnpx serverless deploy -s $STAGE
cd -
To run (rerun) the post-deployment steps:
cd main/solution/post-deployment
pnpx serverless invoke -f postDeployment -s $STAGE
cd -
To re-deploy the UI
cd main/solution/ui
pnpx serverless package-ui --stage $STAGE --local=true
pnpx serverless package-ui --stage $STAGE
pnpx serverless deploy-ui --stage $STAGE --invalidate-cache=true
cd -
Note: These are optional steps.
To view information about the deployed components (e.g. CloudFront URL, root password), run the
following, where [stage]
is the name of the environment (defaults to $STAGE
if not provided):
scripts/get-info.sh [stage]
Once you have deployed the app and the UI, you can start developing locally on your computer. You will be running a local server that uses the same lambda functions code. To start local development, run the following commands to run a local server:
cd main/solution/backend
pnpx serverless offline -s $STAGE
cd -
Then, in a separate terminal, run the following commands to start the ui server and open up a browser:
cd main/solution/ui
pnpx serverless start-ui -s $STAGE
cd -
For more information, refer to Service Workbench Installation Guide.
Once Service Workbench is fully deployed, the console will output the Website URL and Root Password for Service Workbench. You can log in by navigating to the Website URL in any browser, and then using the username root and the Root Password given by the console. Please note that logging as the root user is highly discouraged, and should only be used for initial setup. You can create a new user by clicking the Users tab on the left, then Add Local User. Follow the instructions given to create the user (you can leave the Project field blank for now), then log out of the root account and into your new user account.
Adding a local user should only be done in test environments. We highly recommend using an IDP for prod environments. For more details on how to set up an IDP, click here
Once in your user account, you'll need to link your AWS account. Navigate to AWS Accounts in the left bar, then click the AWS Accounts tab. From here, you can create an AWS account, or link an existing one.
To create a new AWS account, you'll need the Master Role ARN value, which you can get by contacting the owner of your Organization's master account. If you are the owner, you can find it in the Roles section of AWS IAM from the AWS management console.
To link an existing account, follow the instructions listed. You'll need the following credentials:
Now that you have a user and have a working AWS account, we can start generating workspaces. Workspaces allow you to use AWS resources without having to manually set up and configure them. In order to create a workspace, your account has to be associated with a project, which has to be created under an index.
Pre-requisites: Before creating a workspace, you must setup Service Catalog. Refer to the Import a Product section of the Service Workbench Deployment Guide for information on installing Service Catalog.
Your workspace may take some time to launch. Once it is up and running, you can connect to it by choosing Connect. For more details, see the following documentation pages:
go-research-on-aws
. For more information on using EMR Notebooks, see Using EMR Notebooks.To change the default password for Jupyter Notebook instances, contact your Solution Architect, raise an AWS support case, or follow these instructions:
main/solution/machine-images/config/infra/provisioners/provision-hail.sh
:s/sha1:<salt1>:<hash1>/sha1:<salt2>:<hash2>/
<salt2>
and <hash2>
to match your password’s corresponding values.main/solution/machine-images
.pnpx serverless build-image -s <stage>
to create a new AMI for EMR environment types.Note: EMR workspaces are not available if AppStream is enabled for the deployment.
Studies are datasets that you can tell Service Workbench to preload onto your workspaces. When your workspace has finished provisioning, you will immediately have access to any datasets within Studies associated with that workspace.
In the navigation pane, under the Studies tab, choose Create Study. The ID field represents the ID for that particular dataset. Studies can also be associated to projects using the Project ID field. Once the study has been created, you can upload data files with the Upload Files button.
Once you have a study with datafiles loaded, you can start provisioning workspaces with your study data. In the Studies tab, select one or more studies. The data in these studies is preloaded onto the AWS compute platform. In addition to your own studies, you can also choose from your organization's studies and/or open data studies (publicly available datasets). After choosing your desired studies, click Next to create a workspace. Refer to the Workspaces section for documentation on the compute platforms.
Once you have finished determining the properties of your workspace, Service Workbench generates your workspace and preloads it with your study data. You can access it from the Workspaces page by choosing the Connect button on your workspace.
Start by looking at these files:
They are meant to provide a sample service, a sample controller and a sample UI page.
Follow these steps to add a custom Service Catalog product:
addons/addon-base-raas/packages/base-raas-cfn-templates/src/templates/service-catalog
folder.addons/addon-base-raas/packages/base-raas-post-deployment/lib/steps/create-service-catalog-portfolio.js
(lines 23-35).
Note: Line numbers might change in a future release.environment-deploy.sh
script.To audit the installed NPM packages, run the following commands:
cd <root of git repo>
pnpm audit
Please follow prevailing best practices for auditing your NPM dependencies and fixing them as needed.
This project is licensed under the terms of the Apache 2.0 license. See LICENSE. Included AWS Lambda functions are licensed under the MIT-0 license. See LICENSE-LAMBDA.