search

OpenWaterFoundation / owf-app-infomapper-builder-ng

Open Water Foundation web application to build an InfoMapper configuration
GNU General Public License v3.0
0 stars 0 forks source link

AWS S3 - Implement storage for Builder sites #8

Open smalers opened 1 year ago

smalers commented 1 year ago

The Builder functionality is reaching a point where configurations that are created need to be saved to persistent storage. Although it may be possible to store the configurations in any cloud storage platform, the initial focus is AWS, with GitHub being another consideration.

Technical Considerations

The following are technical considerations:

  1. The storage of an InfoMapper configuration should have some flexibility:
    1. Allow the S3 location to be specified when an InfoMapper site is created.
    2. The list of valid OWF S3 hosting sites should be presented to the user. This can currently include OWF buckets/folders that are known to be properly configured (CloudFront, lambda, etc.), and could allow the user to specify their own (we will need to define requirements for hosting).
    3. Does the list include only buckets, or bucket and folder? The latter would be more flexible. For example, I could see sites being deployed for:
      1. Dedicated S3 bucket such as poudre.openwaterfoundation.org
      2. Bucket shared with many implementations, such as infomapper.openwaterfoundation.org/some-infomapper, perhaps with sluggified organization name in the path.
      3. More focused organization such as rivers.openwaterfoundation.org/co/clear-creek/latest/...
  2. Once the S3 location is specified, content will need to be organized:
    1. Name high-level folders with AWS Organizations in mind and authentication.
    2. Someone should not be able to edit another organization's files.
    3. How to publish using a general URL (like the Poudre) rather than an organization-based URL? Maybe there needs to be the concept of a "community" implementation that spans Organizations compared to Organization implementations that contain InfoMappers to support an organization's work? InfoMapper site type could be "Personal", "Organization", or "Community". A Community site will need to allow someone to be assigned to admin/write roles.
  3. CloudFront integration must be considered:
    1. It should be possible to interact with the S3 API directly, but public URLs that hit CloudFront will require invalidating the resources.
    2. Are CloudFront distributions built automatically for site-specific buckets (such as third party hosting of a site) or does OWF host on a bucket that includes multiple implementations and a larger distribution is used?
    3. Need to be aware of CloudFront fees, such as invalidations across the entire business.
    4. Let's start small and figure out the mechanics but plan to scale.

S3 Folder Conventions for an InfoMapper

The following use case illustrates how the InfoMapper folders could be named.

  1. Authenticate - this information is separate from any InfoMapper.
  2. Start a new InfoMapper project. Pick the S3 hosting location from a known list. Based on the the type (Community, Personal, Organization), may have different restrictions and conventions.
    1. Maybe known hosting locations have parameters that indicate what types of sites (Community, Personal, and Organization) can be hosted?
    2. Does setting up a Community site require OWF staff to initialize? Maybe so.
    3. For Personal and Organizational, top level folder in hosting bucket/folder is a sluggified account name.
  3. If an existing project, pick from a project and version. Otherwise, create a new project and the version will be 1.0.0, for example. The corresponding folder will be used to store the files in S3. The following table summarizes the top-level folder names.
  4. Within a top-level folder exists a normal Angular application with index.html, application bundles, and assets.
Folder Description
1.2.3 Versioned folder that holds a complete working application and can be archived for as long as needed. This is the starting point of the next iteration/version. Versions allow keeping an archive for some period of time and allow reverting the latest version if necessary.
preview A folder where the current working files are being edited and can be previewed, prior to publishing as latest. This is a throw-away copy. A web browser can be pointed to this during development.
latest A copy of the latest 1.2.3 versioned folder, which is intended for public use. This should always be an exact copy of the most recent accepted versioned folder.