ised-isde-canada / cbmdsp-cp-sp-overall-architecture

1 stars 1 forks source link

Epic: Meeting Agendas #6

Open obriensystems opened 3 years ago

obriensystems commented 3 years ago

Links

Internal WIKI: https://wiki.ised-isde.canada.ca/display/DTSSBLSD/AWS+Event+Driven+Architecture#AWSEventDrivenArchitecture-MeetingMinutes https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/projects/1?add_cards_query=is%3Aopen

https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/wiki

20210617:1500EDT

Agenda

1) arch review pre TRB submit this friday for 23rd

2) Services tasks

Context around group ownership of services (AWS only, OCP only, CoE provided) - or more around which services require direct collaboration with Artur's team

services supported (dynamodb, keycloak, s3 from ocp) services not yet supported - requiring guidelines (api-gateway, ses, s3 from public, security additions for unsupported services

go over interfaces list

3) quick sced review - in the context of incoming https connection not going through the OCP ALB

4) jira board task creation

5) github repo layout for wiki/docs/pocs

https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/2

https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/wiki

6) review the scope of the 2 thu/fri planning/dev meet scopes

(Thursday meeting is optional for all those in the optional CC list - the meeting is focused on task planning, high level architecture and project management - not detailed dev issues)

(Friday meeting is a pure hands on dev meeting and is fully optional for all levels of management - where we will work out dev/commit/environment issues)

Rob Carter for AWS ingres security beyond SCED/SEA - if we need it even for PUTs

Stephane: WLM required for NUIX applications on the older EDS when we move to end state EDC OCP cluster on prem current injestion is internet ready to do GETs from S3 from on-prem

cc as optional 15 min at 230 fri - send out cancel 24th SJBD holiday send mail on TRB results from 23 for 24th

obriensystems commented 3 years ago

20210625:

Stage 3 TRB approval was OK on the 23rd we are proceeding with stage 4
go over stage 4 details
go over work items
24
20
27
22
8
9

General dev work around validating all interfaces in arch diagram https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/wiki/Architecture
review schedule next 2 weeks (vacation...)
Other teams implementing serverless (apigw+lambda) = Rabih Raffoul

actions deck to Carina (pending approval)

verify issue linking /blocked by in github Rob Carter on 13 sec add vsd to arch github add sec task for Rob move sec rails to 8

convert subtasks to tasks put back to epic

ask about preprod

July 27 IEOC present by thomas

obriensystems commented 3 years ago

20210708 Agenda

High Level

continue with stage 4 prep after stage 3 approval including document close off for 27 July MVP formal creation in repo - to drive testing prep for UAT (MINU intro testing) security review architecture validation/iteration
iterate use cases for External and MINU user focus - include (UC, state, sequence diagrams)

Links Scrum Board https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/projects/1 Wiki https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/wiki/Architecture

Review meetings

Details

Stage 3 TRB approved 23 June - next milestone is the IEOC presentation by Thomas on 27 July
SCED pilot mid Sept 2021 - we should come in with the same type of flow after that app
correlating github tasks with jiras - expanding on aws work item subset
https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues
https://jira.ised-isde.canada.ca/projects/CBMDSP/issues
Moving all the prototyping infrastructure/code (lambda/api-gateway/s3/dynamodb/(ec2 and/or container testing code) into cloudformation templates - for reproducability and verifying the trusted advisor rules on the account for example S3 bucket logging changed yesterday https://console.aws.amazon.com/trustedadvisor/home?region=ca-central-1#/category/fault-tolerance?checkId=BueAdJ7NrP
Assist with project template specific to AWS work
complete security guardrails docs - send out 1-4 of the 13 as preview
Collaborating with the AWS SEA team (ISED implements SEA) - https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/25
We are on 1.3.4 of the SEA
complete stage 4 requirement doc in teams - moved to final (one final edit)
Start working out MVP, test scenarios to drive use case iteration
S3 throughput testing from inside the account - from 225-600mbps using a T3.xlarge EC2 in the same region - retrying with 20gbps networking
https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/29
getting extra bell line for 200mbps external testing
simulate external client via Azure vm rest calls
cloudfront distribution - OK except hander for encrypted bucket
https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/24
WI10:CloudFrontPresignedHostingofS3privatebucketsite
AWS SAM integration with CF Brian: use secondary cfncli https://github.com/Kotaimen/awscfncli - will transition to from straight CF (s3 push included) Dev lifecycle needs to be improved outside ISED https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/32

obriensystems commented 3 years ago

20210715 Agenda

last week review (agenda) home testing throughput: Rogers + Bell = 30 + 10 = 40mbps via dual-wan router - starlink in queue http://wiki.obrienlabs.cloud/display/DEV/Internet+Access
Architecture direction Lucidchart visio exports in https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/tree/master/documents

9

1) moving towards dual client for s3 upload (js + cli) - or working around the presigned key/file-piece + throttling issue (browser based : html/js for small workloads - presigned dynamic creds) - see Brian's link to https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/2#issuecomment-858898763 in

2

(CLI based : aws cli for large workloads - use assigned temp creds) - cli will be packaged in a runnable app Work out the impact on the UI experience with a split browser/app
use cases review' detail multipliers x 2 (volume, error, type of client, split on 2 users = ~ 8-12 for now)

7

start UI framework on existing cloudfront S3 artifacts in #24 move previous Amplify page to subfolder https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/blob/master/lambda.html
work items review original list in https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/2#issuecomment-858898763 sftp testing throughput testing setup dynamodb schema #12
test cases review Use case flow (MVP end to end) - push, process/validate, notify, pull, delete Volumetrics (in-region max pull - test CLI SDK concurrency/retry on large/wide uploads) Resiliency (validation during and after (MD5) including counsel request to retry)
security review Single security-only version of the arch doc as TOC for 13 point sec review extracting local doc to #25 and prior to review
poc/mvp review in progress move from manual console artifacts in s3, cloudfront distribution, api-gateway, lambda, dynamodb, iam execution roles, https certificate manager, route53 records, vpc private subnet, vpc endpoints (gateway in TG prod) - into cloudformation templates
devops review We (CB SP) are still in the SCED queue along with other WLM projects - there is a SSC common arch capture process on the go - our arch is being consumed as part of a spanning tree coverage of "all" ised arch attributes/flows account access requests (aws or ocp) switch to java 11 from 1.8 for sam/cfncli support push SAM API stub in #5 installed CFNAPI - pushing stub in #33 cfnapi can wrap samapi
plan for next week (agenda)
Dates SCED project 1 (we are next in the queue) TG flows should be in by mid Sept (up from end of Aug) - WLM meet
Additional

email on assigning All, History on assigning to tasks in github “discussion at the end of our meet” – normally you can auto-assign yourself (I can) – but there are issues assigning Brian to start – I assigned one task to him – was not able to assign any other under “BrianFanning” Anyway, sending mail out on history this one time to the team In the future any discussion like this will remain on the main meeting task Which shows up on the middle column on the jira board (todo, in-progress, finished) https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/projects/1 as the top #6 jira “Meeting Agendas” https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/6

Brian, I noticed that I could assign the 2nd jira after you commented on it https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/2 – effectively making you a collaborator I will look into why this is with the ISED devops team And why you cannot auto assign So I assigned you to one you commented on earlier #2 – actually now both of us You show up as an “assigning suggestion” after you comment

Summary This one 7 days ago I assigned to you after you commented – lucky https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/20 This one worked because you commented as well in the past https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/2 This one should be yours – after you comment – I will assign https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/18

thank you /michael

obriensystems commented 2 years ago

20210722: Agenda

go over end to end use case flow in #7
go over high level security doc in prep of signing off on the security checklist in #8
introduce Olga Novoa to team - our UX specialist - welcome
any devops items
check self assign for brian for github issues - comment first may fix it

obriensystems commented 2 years ago

20210729: Agenda

security controls docs submitted for Stage 4 meet on the 3rd
security controls docs also going through review by the security team (Michel) - 4 of 13 remaining open
updated ASC-2 - https://jira.ic.gc.ca/browse/ITSECAA-487 ASC-6 - https://jira.ic.gc.ca/browse/ITSECAA-491
working ASC-7 https://jira.ic.gc.ca/browse/ITSECAA-492 ASC-4 https://jira.ic.gc.ca/browse/ITSECAA-489
Thanks Brian for finding about single/multi file upload perf issue - add to devops design (compress/decompress for transport)
20
developer connectivity
- 3 more AWS access connectors for team
- continue to retrofit backend lambdas for partitioned dev + stg (a stable regular release for Olga) The current html/js stub for example is pointing to a WIP api-gateway backend - need to commit stable index.html version pointing to either api-gw/lambda or ocp backend https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/blob/master/lambda.html
- all UX options open thin html client (js or thymeleaf), thick client for SDK/SCP
- continue to move API/Code into the repo including the thymeleaf stub
sonar project local-docker/ised versions
jenkins project - local-docker version for non-ised
discuss the supplied ALB with default routes for OCP to subnet comms

Calls the ALB with the default 443 target group pointing to the stub lambda function https://ca-central-1.console.aws.amazon.com/lambda/home?region=ca-central-1#/functions/PBMMAccel-DevCbCbMdsp-Pha-ElbLambdaFunctionCorehea-882jgUVRyNSc?tab=code

C:\WINDOWS\system32>kubectl exec -it backend-stub-68f6d45d58-278p4 bash [I have no name!@backend-stub-68f6d45d58-278p4 ~]$ curl https://internal-core-devcbcbmdsp-alb-1684387230.ca-central-1.elb.amazonaws.com/ --insecure Hello from: ***0611

SCED SEA code to deployment correspondence - IE: default ALB in account and default lambda target

https://github.com/aws-samples/aws-secure-environment-accelerator/blob/main/src/deployments/cdk/src/deployments/alb/artifacts/internal-dev-alb-lambda.txt

https://ca-central-1.console.aws.amazon.com/lambda/home?region=ca-central-1#/functions/PBMMAccel-DevCbCbMdsp-Pha-ElbLambdaFunctionCorehea-882jgUVRyNSc?tab=code

discuss SEA upgrade affecting manual/dev mods of SEA generated resources - go over the ALB
possibly ask for r/o access to CIO AWS codecommit repo - to assist in rev-eng of the deployment arch example: https://ca-central-1.console.aws.amazon.com/codesuite/codecommit/repositories/infrastructure/commit/b8b78538d1ea2d99c47a3d9b70119987fd89b908
sprint 0 starting next week after planning session https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-database-instance.html

todo

Robert meetings, jiras
raised teams message/jira for AWS SEA PBMM template access for deployment diffs/reverse engineering for hybrid envs https://jira.ised-isde.canada.ca/browse/COPS-8240 https://jira.ised-isde.canada.ca/browse/CBMDSP-40
dual ocp + aws dev pipeline proto (CodePipeline for CodeBuild/CodeDeploy) https://jira.ised-isde.canada.ca/browse/CBMDSP-41

obriensystems commented 2 years ago

20210805: Agenda

Thomas: IEOC ( Investment & Experimentation Oversight Committee ) stage 4 review on the 3rd Aug
security controls docs submitted for Stage 4 10 of 13 reviewed
Stage 4 planning started on 5 Aug, continues 9 Aug
project plan talk
Work items review
https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/projects/1
prepping env for UX design - all of amplify/cloudFront/thymeleaf(OCP)
reduce DEV meet fri 1430-1530 meet to 30 min to 1500 as we collab at this thu meet and via github

Todo:

scrum level round/table in next week? y around https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/projects/1
Note: to assign jiras/tasks - comment on the jira first - then you can be assigned unless you originated it

Evolving volumetric test data (all public EC radar/sat/vaisala images for now) - #37

public counsel/client testing data in S3 - us-east-1 for use by EC2...(on/off region, on/off aws) -- 1.4g / 12K files https://upload-weather-lightning.s3.amazonaws.com/radar_lightning-ATL.tgz -- 3.5g / 11K files (under 5) https://upload-weather-lightning.s3.amazonaws.com/radar_lightning-CLDN.tgz -- 10g (over 5) 51K files https://upload-weather-lightning.s3.amazonaws.com/radar_wcan.tgz -- 253g (300g) 1.01M files https://upload-weather-lightning.s3.amazonaws.com/tgz.zip

obriensystems commented 2 years ago

20210812 Meeting

repo for post stage 3 work moved to bitbucket - sonar/jenkins updated https://wiki.ised-isde.canada.ca/display/DTSSBLSD/SP+Developer+Guide#SPDeveloperGuide-Quickstart https://jira.ised-isde.canada.ca/browse/COPS-8367
we will keep this github repo for AWS PS code sharing
JIRA Epic/Stories review for sprint 0 exercises this week
Review Brian's proposal for the managed upload script #18
welcome with Christabel Kim
cancel this fridays dev meet at 230 - need to cool down shoulder

Ongoing

get team capacity, create story tasks, size stories
interleave/meet-in-the-middle on jiras from 1-50 representing arch/tech with 50+ representing blackbox UX centric jiras
start UX screen mockups, iterate into MVP
start formal MVP - pick success flow (minimum path, maximum spanning tree)
refs #2
20
19
integrate snapshots of prototype code from github to bitbucket - TBD modify source in github until each release - or reverse cherry pick - to developer discuss
devops check (accounts, access....)

1535 todo: very good demo by Brian - we will review the code and run it from github - thank you get project plan extract to AWS SA Christabel replacing Carina Christabel to send github username to michael for project setup

obriensystems commented 2 years ago

20210819: meet

Until 2 weeks ago we were in architecture, proof of concept, interface validation (which services viable) - with a more technology/deployment focus to work through stage 2-3
In stage 4 we first step back into UX and behavior -
we are still in discovery mode as part of discovery - pre sprint 1 on 22 Sept (build phase after design phase)
epics and stories created - working on task discovery
iterate until delivery/design starts (tomorrow until 22nd Sept) - where we can start assigning additional stories on top of the UX ones
present focus is UX oriented tasks driving an E2E MVP flow - wireframe for now -- specifically field level attributes (lists, dropdowns, predefined/freeform fields, buttons) - in reproduction of original forms/process -- in addition to behavioral - updating activity diagrams
researching/reviewing original requirements, business process flow pdf
Some of the form data is public domain - evolving list around
https://www.competitionbureau.gc.ca/eic/site/cb-bc.nsf/eng/01705.html https://www.competitionbureau.gc.ca/eic/site/cb-bc.nsf/vwapj/Notifiable-Transactions-Regulations-form-e.pdf/$FILE/Notifiable-Transactions-Regulations-form-e.pdf verify certificate needed in https://www.competitionbureau.gc.ca/eic/site/cb-bc.nsf/vwapj/Certificate-en.pdf/$FILE/Certificate-en.pdf
review Brian's https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/pull/38
review last week
plans for next week

review visio diagram edit/sharing first via wiki, ideally via teams https://techcommunity.microsoft.com/t5/visio-blog/visio-as-a-tab-in-microsoft-teams-is-generally-available/ba-p/1474787 upcoming adobe wireframe demo of how the tool works by Olga

obriensystems commented 2 years ago

20210826 Meet

Agenda

We are in the first week of a 2 week Design as part of Delivery phase (post discovery but pre Build) from (1 week late) 23 Aug - 03 Sept
Review JIRA board for the first 3 of 6 Epics
https://jira.ised-isde.canada.ca/secure/RapidBoard.jspa?rapidView=205&projectKey=CBMDSP&selectedIssue=CBMDSP-99
https://jira.ised-isde.canada.ca/browse/CBMDSP-51 to 54
Review per-epic stories around 4 tiers - see https://wiki.ised-isde.canada.ca/display/DTSSBLSD/SP+Developer+Guide#SPDeveloperGuide-Design -- UX (Wireframes driving frontend with stubbed backend calls) -- Front end (TBD: thymeleaf served via spring boot OCP container) -- Back end REST (OCP except for upload endpoints) -- DB (dynamoDB)
Review AWS PS engagement on upload Epic as it progresses to the point where we use the client/backend POC as a base
Group edit of ongoing activity/flow diagram https://wiki.ised-isde.canada.ca/display/DTSSBLSD/SP+Architecture#SPArchitecture-ActivityDiagrams

Notes

Action item: Discuss with Artur whether we can use STS integration with Cognito via Keycloak - (aws temp creds) - Brian to investigation

obriensystems commented 2 years ago

20210902: AWS Meet

https://wiki.ised-isde.canada.ca/display/DTSSBLSD/SP+Developer+Guide#SPDeveloperGuide-20210902:AWSMeet-RequirementsReviewFocus

Agenda

Requirements Review / mapping to stories SP Architecture#RequirementsAnalysis Questions for CB on requirements - list to be posted Mention Brian status change Plans for next week

Answer to location of AWS PS stories

Christabel,

Hi, in answer to the outstanding stories and future work. The following tag on the github jiras shows 3 tasks, the first 2 are the in progress thick(sftp+) and thin(browser) client support for AWS upload API’s being worked on by Brian.

The last one is an earlier – scoping jira on possible pieces of the arch that we may need assistance on. I expect the number to expand as we go through the MVP.

https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues?q=is%3Aissue+is%3Aopen+label%3AAWS

obriensystems commented 2 years ago

20210909: AWS Meet

https://wiki.ised-isde.canada.ca/display/DTSSBLSD/SP+Developer+Guide#SPDeveloperGuide-20210909:ThuSPMeet Integration of Pierre's sitemap diagram in teams Finish splitting of the flow diagram SP Architecture#ConceptualWebsite-Sitemap

in visio https://teams.microsoft.com/l/file/7348FD6E-8B36-4C06-87C0-B8DD3F10AB52?tenantId=b72ac62f-06d5-4cd5-824e-ee92319a4676&fileType=vsdx&objectUrl=https%3A%2F%2F033gc.sharepoint.com%2Fsites%2FSubmissionPortal%2FShared%20Documents%2FGeneral%2FStage%204%2FConceptualWebSite.vsdx&baseUrl=https%3A%2F%2F033gc.sharepoint.com%2Fsites%2FSubmissionPortal&serviceName=teams&threadId=19:17b38c8b7afc4243a120dc92de8fc027@thread.tacv2&groupId=2c9bfa66-f86d-4066-92e3-59653e90e3b3 Finish off requirements review SP Architecture#DiscussionandQuestionPointsonRequirements finalize schema for status entity finalize schema for home page entities Finish end-end demo "home page" flow for read only query of preset dynamodb data, orm connector, rest read controller, thymeleaf template translation of ux extract 3 of 11 security guardrails documents discuss stage 4 draft docs

Start integrating presigned url post code into submission flow cover off cognito/keycloak feasibility question on federation discuss milestones coming up June https://teams.microsoft.com/l/file/46283DF0-0FD2-499F-BFEB-97B1493EE48B?tenantId=b72ac62f-06d5-4cd5-824e-ee92319a4676&fileType=pdf&objectUrl=https%3A%2F%2F033gc.sharepoint.com%2Fsites%2FSubmissionPortal%2FShared%20Documents%2FGeneral%2FStage%204%2FCB%20SP%20Plan.pdf&baseUrl=https%3A%2F%2F033gc.sharepoint.com%2Fsites%2FSubmissionPortal&serviceName=teams&threadId=19:17b38c8b7afc4243a120dc92de8fc027@thread.tacv2&groupId=2c9bfa66-f86d-4066-92e3-59653e90e3b3 Aug https://teams.microsoft.com/l/file/AD756CC6-C09D-4C1E-892C-335A16CB04A0?tenantId=b72ac62f-06d5-4cd5-824e-ee92319a4676&fileType=pdf&objectUrl=https%3A%2F%2F033gc.sharepoint.com%2Fsites%2FSubmissionPortal%2FShared%20Documents%2FGeneral%2FStage%204%2FCB%20SP%20Plan%20Timeline%202021-08-13.pdf&baseUrl=https%3A%2F%2F033gc.sharepoint.com%2Fsites%2FSubmissionPortal&serviceName=teams&threadId=19:17b38c8b7afc4243a120dc92de8fc027@thread.tacv2&groupId=2c9bfa66-f86d-4066-92e3-59653e90e3b3 TRB stage 4 draft - revisit https://teams.microsoft.com/l/file/292B1002-0754-4184-8534-FABAEE16118A?tenantId=b72ac62f-06d5-4cd5-824e-ee92319a4676&fileType=pptx&objectUrl=https%3A%2F%2F033gc.sharepoint.com%2Fsites%2FSubmissionPortal%2FShared%20Documents%2FGeneral%2FStage%204%2Fdraft%2F4S%20Tier%203%20Technical%20Review%20Board%20Presentation%20-%20CB%20Mergers%20Directorate%20Submission%20Portal%20Project%20-%20PRJ0001OP.pptx&baseUrl=https%3A%2F%2F033gc.sharepoint.com%2Fsites%2FSubmissionPortal&serviceName=teams&threadId=19:17b38c8b7afc4243a120dc92de8fc027@thread.tacv2&groupId=2c9bfa66-f86d-4066-92e3-59653e90e3b3 review UC around home page epic 5 levels of subtask around main landing page

including a separate CD testing task (on top of normal CI/Junit/mockito) - on a deployed system

UX Thymeleaf page rendering (home.html, css with TL tags) + HomeController REST API backend (Service/API) - including ORM to JAXB (db to client POJO) mapping DynamoDB repository CD Test cases

Dependent on overall:

DynamoDB schema DynamoDB ORM/repository framework CD testing framework

Prep for dev meet 1430 Friday create 3Scale proxy url request for OCP container discuss handcoded for initial HomeController or OpenAPI schema/controller/API generator - 1 time stub or bidirectional regeneration directly to non-editable target/source - essentially all schema/REST api up front Continue what Pulak started - a persistent OCP service url to use as a base for the OCP 3Scale reverse proxy (short URL) - can be DeploymentConfig or Deployment + service k8s native yaml - https://scm.ised-isde.canada.ca/projects/ICAPPS/repos/cb-cbmdsp/browse/src/kubernetes/service.yaml

Continue Pulak's CD starter framework around OCP pod redeploy on image registry upload form master build - we should either add a CD minimum spanning tree short regression test to validate the image - or keep a 2nd container up as a validated working demo - Michael todo: finish moving subtasks off hoild Cover off availability to AWS Pro Serv confluence - no? Bitbucket - yes github - yes - propose abandon repo if AWS can merge directly to Bitbucket - to ease double merge Propose selected diagrams/arch in the readme https://scm.ised-isde.canada.ca/projects/ICAPPS/repos/cb-cbmdsp/browse/README.md Fixed AWS S3 non-versioning bucket - https://isedaws.awsapps.com/start/#/

obriensystems commented 2 years ago

20211007: meet

review board review AWS PS requirements New SA github account work items - github link https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/labels/AWS access to AWS account access to OCP account - https://console-openshift-console.apps.ocp.dev.ised-isde.canada.ca/topology/ns/cb-cbmdsp-dev?view=graph https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/6 plan for sprint 2 moving WIP tasks over plan for MVP 1 - 1st week Nov plan for UAT

older mail https://wiki.ised-isde.canada.ca/display/DTSSBLSD/SP+Developer+Guide#SPDeveloperGuide-AWSProfessionalServicesPersonnelCollaborationInviteMail

Sounds good, welcome.

I would require <dev>’s github username (personal or corporate – I use my personal) to add to the shared project we have on github

Verify that 2FA is enabled on the account – required

https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture

AWS SA label/tag

https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues?q=is%3Aissue+is%3Aopen+label%3AAWS

AWS SA potential items

https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/2

AWS SA meeting minutes

https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/6

Optionally, depending on how much is already integrated with ISED the following would be useful – let me know if you have any of these.

0 – cloud wiki (optional – as we can duplicate select sections around work items to github)

https://wiki.ised-isde.canada.ca/display/DTSSBLSD/SP+Developer+Guide

1 – cloud bitbucket access (our main design repo is on cloud bitbucket) - optional

https://scm.ised-isde.canada.ca/scm/icapps/cb-cbmdsp-dev.git

2 – cloud Jira access (optional) – we can use the issues section of github

https://jira.ised-isde.canada.ca/secure/RapidBoard.jspa?rapidView=205&projectKey=CBMDSP

3 – cloud OCP access via RBAC from the github project

https://console-openshift-console.apps.ocp.dev.ised-isde.canada.ca/topology/ns/cb-cbmdsp-dev?view=graph

via

https://github.com/ised-ocp-dev

4 - AWS cloud account – you can use your own as well – as we are still mostly application agnostic

  I would stick to using your own account for experimentation – as the following dev account is very locked down by the trusted advisor rules to close to production level.

https://isedaws.awsapps.com/start/#/

reference

The AWS SEA we currently use to deploy the perimeter and application (CB) accounts/VPCs around the 2 TG’s https://github.com/aws-samples/aws-secure-environment-accelerator

optional

Jenkins https://cicd.ised-isde.canada.ca/job/cbmdsp/

Sonar https://sonarqube.ised-isde.canada.ca/dashboard?id=ca.canada.ised-isde.dev.ocp.apps%3Acbmdsp

Registry https://console-openshift-console.apps.ocp.dev.ised-isde.canada.ca/k8s/ns/ised-ci/imagestreamtags?name=cbmdsp

obriensystems commented 2 years ago

20211014: Meet SP/AWS Sync

Agenda

Notes uat aws account Wenli IDM/Keycloak demo - see users in CBMDSP-5 - IDM Keycloak integration OPEN sprint 2 timeline Timeline Q super user -"20211013: update - minu is the super user - no need for 4th personna" - CBMDSP-69 - As a MINU (super user) I can manage Counsel/MINU/CB-Mergers-staff registration via portal OPEN Review AWS Dev account procedures - thanks Pierre Roles and Responsibilities - For dedicated AWS Account holders take off Brian, Bipan from aws/ocp meet with Yapeng on size/details of workload friday 930 reschedule meet at 230 with sp devs

AWS SA: Yepeng notes 20211015

Yepeng can work 15-20h per week on SP formal requirement for the next 2 weeks filesize (try for 10 but ideally just do 4 for ease of testing) is for example over 10 GB which will take around 45 min to upload (if we see a 30 min html session timeout - reduce to under 30min of file upload = 4GB you can assume all api-gw, s3, lambda, iam setup on your own AWS account for now you can merge all your PRs immediately after posting them - for offline review by the team here - but no blocking on code submits on your end for velocity

priority 1 thin js using backend python https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/43 Assume prerequisites (like IAM assigned bucket - for signing is done) - as we iterate we will prioritize/assign the work items - liase with CIO team COPS tickets IAM role for S3 buckethttps://s3.console.aws.amazon.com/s3/buckets/cb-sp-dynamic-upload-test?region=ca-central-1&tab=permissions in Lambda https://ca-central-1.console.aws.amazon.com/iam/home#/roles/cb_sp_portalGenerateSignedURLviaAPIGWPython-role-piui2kgm?section=permissions api gateway front for s3 https://ca-central-1.console.aws.amazon.com/apigateway/home?region=ca-central-1#/apis/e0r25oiut3/resources/nanyxkkxbl lambda references predefined s3 bucket https://ca-central-1.console.aws.amazon.com/lambda/home?region=ca-central-1#/functions/cb_sp_portalGenerateSignedURLviaAPIGWPython?tab=configure as see also http://wiki.obrienlabs.cloud/display/DEV/AWS+S3+Use+Cases

from https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/generateSignedURLviaAPIGWPython?tab=code

API-GW: https://ca-central-1.console.aws.amazon.com/apigateway/home?region=ca-central-1#/apis/e0r25oiut3/resources/nanyxkkxbl

Hosting: http://portals3.cloudlift.systems/lambda.html

URL: https://kvzryfi6hc.execute-api.us-west-2.amazonaws.com/test

project: https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture

import uuid import boto3

def lambda_handler(event, context):

Get the service client.

s3 = boto3.client('s3')
db = boto3.client('dynamodb')

# Generate a random S3 key name
upload_key = uuid.uuid4().hex

# Generate the presigned URL for put requests
presigned_url = s3.generate_presigned_url(
    ClientMethod='put_object',
    Params={
        'Bucket': 'cb-sp-dynamic-upload-test',
        'Key': upload_key
    }
)

data = db.get_item(
  TableName='eventstream',
  Key={"timestamp": {"S":"1002"}, "label": {"S":"auto"}}
)
print(data);

# Return the presigned URL
return {
    "upload_url": presigned_url
}

For now assume the API Gateway/lambda execution pair are in the public subnet (no private VPC yet) - we will deal with calling the function from OCP in the future - for now direct from the html/js client

priority 2 thick python cli using backend python https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/44 priority 3 (SFTP client) - only after thick/thin clients above are running in MVP 1/2 https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/20

obriensystems commented 2 years ago

20211021:1500 SP meet Agenda

https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/issues/6

MVP1 pre demo for 1st Nov status End to end html (served via thymeleaf + temporary ajax js) calls APIGW/lambda GET for presigned URL returns presigned url json url ajax js calls S3 PUT on single file (will use full multipart js lib from AWS in next demo) optional APIGW/Lambda to close off multipart (future) - for now write metadata directly to dynamodb optional-better: S3 trigger to lambda to write metadata to dynamodb same S3 lambda trigger to SES email Upload screen refresh/ajax shows updated list (1 for now) of S3 file metadata Optional: s3 download for MINU add bucket/folder, counsel id(for dynamodb) as parameters above Optional: actual Cognito/keycloak security - for now IAM role/policy on lambda code and S3 Work items Backend = 145-154 off 54: upload DI1:S3PreSignedURLforS3UploadviaprotectedAPIGatewayendpoint Note: see DI above DI1:S3PreSignedURLforS3UploadviaprotectedAPIGatewayendpoint having denial issues - checking with CIO on whitelist - for S3 PUT note: personal account OK (quick screenshare) Demos UX Start UAT handover in pieces via CF - see Questions Olga: alert for uploaded complete - option clickable

Olga: selecting merger to attach uploading files to - 2 options (one in the merger, the other with "selectable" merger upload backend call will get the folder=merger-id the upload is tagged on - should be no issues olga to review Olga: initiate registration button - send counsel outside of the portal? not recommended but IDM sends the email (out of band) - check with Wenli counsel receives the homepage Olga: Use PDF upload or html page for Counsel start merger certificate forms? https://xd.adobe.com/view/181616b1-8bdf-4ab1-b6fc-2e95c9d6bb65-4d45/screen/a27883b0-6ad4-4a3d-b918-5f4ad4c7f169 will verify with Thomas Notes: Access: check your AWS access Yepeng demo of multipart (single threaded) S3 upload js client retrofit - we can use it in the 1st nov demo, ise the parallel version later merging for https://github.com/ised-isde-canada/cbmdsp-cp-sp-overall-architecture/commit/1f062ff5ea3da8c3d76f350572074af329223a22 See client feedback possible on each return call during the split/upload/finish