Make Open Search and s3 Optional

[ng-druid]

This has been fully implemented for panel pages on both dev and prod demo site and switched to druidcloud.dev and druidcloud.io as well. Panel pages no longer require requests to be used instead they are compiled into the source and loaded into idb when the app starts up. This effort further reinforces rtc / druid dedication to constant sustainable design improvement.

---

Goals:

• Decouple druid from open search and s3
• Facilitate automation of build

Panel pages are currently stored inside open search. The platform use to query open search for route discovery. This has since been replaced by hard coding routes into the application itself inside environment/panelpages.ts and app.module file. However, this does not work when new panel pages are added without manually adding the panel page partial with the route to that file. Furthermore, panel pages JSON should be stored on a more eco friendly solution. Either perhaps inside a github repo, ms cdn, etc. This is a pre-requisite for net0web since once done panel pages can be added and packaged with the app without needing open search or s3. The panelpages and media assets will live in the same repo build repo.

Examples environments/panelpages.ts (ipe)

// Take note args used in demo paths can not be included here.
export const panelpages = [
  ['63a4219d-254e-11ec-ab14-c613312e594f', '/dev-test-virtual-list-flex-v1'],
  ['bb66aa62-8635-4015-86c9-ba4f208b0856', '/native_forms_rebuild_v1'],
  ['a04c3022-cc94-4db0-826b-258efde4abab', '/just-a-lonely-snippet-v1'],
  ['3c3f243f-0ccc-4afc-869d-c37c361cd8fc', '/workflow-designer-v2'],
  ['c720b59b-a26f-4919-bab1-219c358b5b68', '/tractorbeam-test-v3']
];

App module routing

const routes = [
  ...panelpages.map(([id, path]) =>  ({ matcher: createEditMatcher(new PanelPage({ id, layoutType: '', displayType: '', gridItems: [], panels: [], layoutSetting: undefined, rowSettings: [], path })), component: EditPanelPageComponent, data: { panelPageListItem: new PanelPage({ id, layoutType: '', displayType: '', gridItems: [], panels: [], layoutSetting: undefined, rowSettings: [], path }) } })),
  ...panelpages.map(([id, path]) =>  ({ matcher: createMatcher(new PanelPage({ id, layoutType: '', displayType: '', gridItems: [], panels: [], layoutSetting: undefined, rowSettings: [], path })), component: PanelPageRouterComponent, data: { panelPageListItem: new PanelPage({ id, layoutType: '', displayType: '', gridItems: [], panels: [], layoutSetting: undefined, rowSettings: [], path }) } }))
];

Workflow

Save

1. User Saves Panel Page
2. Panel page persisted to Github as new commit using graph QL via vert-go

Fetch

Local

1. Fetch from remote probably using Github repo uri directly to asset

Remote

• Entities embedded in app and compile into source within build.
• No remote fetch needed since data will be in memory. This completely eliminates the need for http requests.

Optional

• Encryption of data on github.

Architecture

• Create a separate repo dedicated to a druids panel pages and other entities.
• Create a verti-go end-point to persist panel pages and/or other entities to github repo easily.
• Create Crud Adaptor to read from local array of in-memory JSON data.

POST /panelpage

The current logic will be swabbed out with the github persistence.

Snippets

This is an example mutation to add/update a file inside a repo using the github graph QL api.

https://docs.github.com/en/graphql/overview/explorer

mutation($input: CreateCommitOnBranchInput!) {
  createCommitOnBranch(input: $input) {
    commit { 
      url
    }
  }
}

{
    "input": {
    "branch": {
      "repositoryNameWithOwner": "rollthecloudinc/ipe",
      "branchName": "master"
    },
    "message": {"headline": "Hello from GraphQL!" },
    "fileChanges": {
      "additions": [{
        "path": "projects/spear/src/assets/GraphQL.md",
        "contents": "aGVsbG8="
      }]
    },
    "expectedHeadOid": "87e0c5e5c62721b83555010ae15f2aa99b390aca"
    }  
}

Requirements:

• content: base64 contents
• expectedHeadOid: expected head before commit

Considerations:

• Need to use graph api to get head commit hash (expectedHeadOid)
• Need to figure out how to authenticate using a token probably
• Need to figure out how to add this to verti-go
• Consider using separate repo ipe-cdn for assets like panel pages.

Graph QL Libs

• https://github.com/shurcooL/githubv4
• https://github.com/hasura/go-graphql-client
• https://github.com/wundergraph/graphql-go-tools

GraphQL Latest Commit on Branch

{
  repository(owner: "rollthecloudinc", name: "ipe") {
    object(expression: "master") {
      ... on Commit {
        history(first:1) {
          edges {
            node {
              oid
            }
          }
        }
      }
    }
  }
}

Taking It Further

This model can actually be taken further. Using a workflow where changes become new branches from master that immediately are MRed and merged into dev. Than a review process could take place to promote to dev. The review process can include notifying stakeholders of the page of something. Allowing them to review changes before promoting from dev environment to prod.

I think this can all be achieved using the graph ql api.

1. Create branch from master
2. Apply changes and commit to new branch
3. Immediately MR and approve into dev branch
4. After period of change inactivity notify stakeholders, subscribers, etc. of page changes

Outline process for promoting to test, prod, etc. environments.

Create and merge pull request using graph QL. https://fireship.io/snippets/github-graphql-merge-pull-request/

entity: pullrequest Action: create, merge

Create a new branch using GitHub graph QL.

Entity: branch Actions: create, archive, delete, merge

Directory Structure (CDN)

Add optional function to Github file upload to determine the file path that will be used for the upload.

Create from guuid. ff7bfcd1-05e5-42ac-bb89-666c47f4a3b1

• f
  • f
    • 7
    • b
      • f

GitHub action read and loop files

https://code.dblock.org/2021/09/03/generating-task-matrix-by-looping-over-repo-files-with-github-actions.html

Graph QL Get File Content

---

Create factory inside keyval module for app initializer that can preload idb with provided data. The input args will be the key, id key | ()=>, data array. The app.module will wire up the app initializer with the panbelpages data exported from the panelpages environment module. This same philosphy can be used for many other entities as well. Should also consider a fallback strategy to lazy load data and add to idb when it doesn't exist in idb. This would be a hybrid solution that might work well for much larger data sets.

Object Resolution Hierarchy

• github objects
  • assets/objects
  • idb

This could be built into crud. Currently when multiple adaptors exist for an entity op the values are merged. Instead a means to turn them into a data resolution hierarchy could be implemented as an alternative strategy for crud.

Implement an affect for panel pages that adds them to idb when fetched from remote perhaps.

One option is to add fallback option to entity config crud metadata. The fallback option would be used in the case where the adaptor returns empty. Also add option to ignore adaptors at root and only allow them to be fallbacks for other meta configs.

I think the way to achieve this is to sort the observables by priority and use concat take 1 instead of a fork join. Also add a filter that requires at least one value for before observable completes. Just need to think about how to deal with all empty values.

• Created keyval app initializer to populate idb with data prior to kicking off app.
• Added additional strategy to crud data helper using fallback flag to execute adaptors in sequence until value is discovered. Essentially, short circuiting the pipeline once the first entity is discovered.
• Implemented fallback adaptor for panelpages entity metadata config.
  • idb_keyval (primary)
  • rest

---

Loop references

• https://www.starkandwayne.com/blog/bash-for-loop-over-json-array-using-jq/

cat $(echo '["ipe-objects/panelpage/3c3f243f-0ccc-4afc-869d-c37c361cd8fc.json","ipe-objects/panelpage/bb66aa62-8635-4015-86c9-ba4f208b0856.json"]' | jq -c 'join(" ")' | rev | cut -c2-100000 | rev | cut -c2-100000 ) > file.txt

Add delimiters using awk

awk 'FNR==1 && NR!=1 {print ","}{print}'  $(echo '["ipe-objects/panelpage/3c3f243f-0ccc-4afc-869d-c37c361cd8fc.json","ipe-objects/panelpage/bb66aa62-8635-4015-86c9-ba4f208b0856.json"]' | jq -c 'join(" ")' | rev | cut -c2-100000 | rev | cut -c2-100000 )  > panelpages2.json

Begin file with content.

echo -e "export const panelpages = [" > panelpages2.json