Refactor of infrastructure to facilitate access control to services / network security

[JimCircadian]

This is a proper refactor, encompass #33 and #36 as well.

• [x] Update provider to v3 #33
• [x] Create virtual network topology
  • [x] Establish basic NSG for client IP access controls
  • [x] Implement public (directly accessible, to be superceded by gateway subnet) & private subnet
• [x] Refactor secrets
• [x] Refactor storage
  • [x] Add endpoints in public network
• [x] Refactor database
  • [x] Add access endpoint within public network
• [x] Refactor processing / events
  • [x] ...
• [x] Considerations for bringing in GeoAPI and EventProcessor
  • [x] Condense additional storage under the singular account
• [x] Gateway subnet implementation
• [x] Ensure project suffix is usable for "client" installations of ETL, defaults to icenet: removed from diagram
• [x] Sort out app-icenet-pygeoapi application deployment
• [x] Address TODO: topic subscriptions setup
• [x] Deploy infrastructure to dev
  • [x] Provision domain for icenet-emails
  • [x] Update email address configuraton in app-icenet-event-processor
• [x] Grab SAS token and ensure capability to upload to input from admin sources

Tests:

• [x] Verify icenet-forecast-topic event triggers app-icenet-event-processor
  • [x] Verify email sent
• [x] Verify app-icenet-processing processes on input
  • [x] Verify database records inserted into psql-icenet-database
  • [x] Verify event deposited on icenet-processing-topic
  • [x] Verify icenet-processing-topic even triggers app-icenet-event-processor
• [x] Verify app-icenet-pygeoapi can view data from database
  • [x] Run notebook for simplistic test

Basically descended from this equivalent GeoAPI issue

[JimCircadian]

Finally got round to sorting out the docker build process for linux function apps. Reducing the size of the build and finishing remaining tasks now. 😉

[JimCircadian]

After some investigation and testing, decided to ensure things are operational. Microsofts documentation is traditionally too verbose meaning the plethora of different services and SKUs have made it a little puzzling to intuitively suss out which way to go. To save time I'm going to ensure the deployments are working out of the box, then add in the public access restrictions and controls

[JimCircadian]

Dealing with infrastructure changes to get to a functional point, but moving access control and security hardening to a new ticket

[JimCircadian]

Sweet, back in action with an icenet capable docker image...

2023-07-26T11:51:54Z   [Information]   Email send poller status: InProgress
2023-07-26T11:52:04Z   [Information]   Email send poller status: Running
2023-07-26T11:52:14Z   [Information]   Email send poller status: Running
2023-07-26T11:52:24Z   [Information]   Email send poller status: Running
2023-07-26T11:52:24Z   [Information]   Successfully sent the email (operation id: 1cc8cdf8-cf84-4ed9-b985-285d102829ec)
2023-07-26T11:52:24Z   [Information]   Executed 'Functions.EventGridProcessor' (Succeeded, Id=4aab466c-da63-4bde-a255-21be8e727f75, Duration=31196ms)

[JimCircadian]

Validated geoapi is working