search

ised-isde-canada / cbmdsp-cp-sp-overall-architecture

1 stars 1 forks source link

AWS Pro Serv Work Items - EPIC #2

Open obriensystems opened 3 years ago

obriensystems commented 3 years ago

Wiki: https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/wiki

Internal reference - will fill out later 20210607 monday - here https://jira.ised-isde.canada.ca/browse/CBMDSP-4 PM Meet Friday

Work Items https://wiki.ised-isde.canada.ca/display/DTSSBLSD/AWS+Event+Driven+Architecture#AWSEventDrivenArchitecture-WorkItemsModulesbyTeam

Latest arch diagram attached CB SP Arch_detailed_20210610_vsdx.zip

OCP login https://console-openshift-console.apps.ocp.dev.ised-isde.canada.ca/topology/ns/cb-cbmdsp-dev

2 AWS resources (Brian (arch) + Andrew (arch) + Robert C (sec), ) Weekly updates on tasks, financial (end of month) - new meet at 1 pm fri

obriensystems commented 3 years ago

from internal https://wiki.ised-isde.canada.ca/display/DTSSBLSD/AWS+Event+Driven+Architecture#AWSEventDrivenArchitecture-WorkItemsModulesbyTeam

Work Items Modules by Team

This quick table is in addition to the JIRAs and github extracts - the purpose is to track/label which work could go to AWS and which items will require ISED personnel.

For example any intersection with the OCP cluster may involve the CCoE team for provisioning a VPC/Gateway endpoint for example.

Area | ISED \| AWSdriven | OCP links | ID ondiag | priorityplus # | AWS API | add depends column | ID on diagram -- | -- | -- | -- | -- | -- | -- | -- Arch | AWS | Y |   | High1 |   | Architecture review of solution by AWS |   upload | AWS |   |   | 2 Med | S3 Upload | S3 upload meter section in the app |   upload | AWS |   | 1 | High1 | S3 upload | JS library for multipart browser S3 upload - SDK like the JAVA one - get link(priortized by already done in other ised project by AWS SA's)check global accelerator up to 20% improvement + lag reduction |   upload | AWS |   | 3 | Lowest99 | Snowball | see: liability option for CCoE eval - do not use |   upload | AWS |   | 2 | 2 Med | SFTP |   |   Download | ISED |   |   | 20 Med | C2G | Transfer from S3 to on-prem EDC new or oldInvolves SCED collab |     |   |   |   |   |   |   |   Serverlessconnectivity | AWSISED/CCoE | Y (not pure AWS work) | 4 | 5 Med | DynamoDBVPC endpoint | Issue is OCP to Serverless commsAPI and connectivity from OCPISED if CCoE provisions the VPC Gateway Endpoint and Transit Gateway routesRequest provisioning from CCoE team via |   Schema | Any |   | 5 | Low | DynamoDB | metadata json structuredjson for events |   Serverless |   |   | 6 | 6 Med | "" | Same DynamoDB API for lambda |   ServerlessConnectivity | AWSISED/CCoE | Y (not pure AWS work) | 7 | 4 Med | AWS API Gateway | private connectivity from OCP for serverlessDepends on |   Emails | Any |   | 8 | Med | emails | tested SES with custom domain - need CCoE approach - likely MS |   website | AWS |   | 9 | Low | CloudFront | CF portal for hosting static site instead of via Thymeleaf in container |   website | AWS |   | 10 | 3 Med | CloudFront | use CF presigning instead of lambda called S3 presigning from container |   website | AWS | Y | 11 | M | Keycloak | SSO using RHSSO container in OCP cluster - route exposed |   security | ISED/CCoE |   | 12 | M | KMS | from ised but implemented by CB team/AWSUse CCoE team |   sec | ISED/CCoE |   | 13 | M | https cert | same AS ABOVE |  


review ppt

BrianFanning commented 3 years ago

Recap of our discussion earlier today on S3 uploading:

While the Javascript AWS SDK does have a ManagedUpload function, it does not support using pre-signed URLs. If you want to use pre-signed URLs in combination with multipart upload, you must generate a unique URL for each part of the upload. There is an extensive thread on the topic here on the SDKs GitHub page.

Multipart uploading with pre-signed URLs can still be accomplished. See the below diagram for how the process would work. Competition Bureau - S3 Multipart upload w_ presigned URL

Notes on the above process:

If we cannot use pre-signed URLs at all due to ITSec guidelines, there are JS libraries available that provide multipart upload-style functionality along with error handling / retries and concurrency, such as flow.js We would need to rebuild the file on the server side on a machine with enough disk space to fully reconstruct the file, and then push the file to S3.

obriensystems commented 3 years ago

Testing assignment after reference @BrianFanning in #18