Build `if-merge`

[jmcook1186]

what a command line tool that takes multiple output files and merges them into one single file according to some simple rules.

why this opens the potential for systems to generate their own manifests independently, then later merge all the manifests from the different components together for time-syncing, aggregation and reporting purposes.

context Let's say we have a set of servers whose impacts we want to measure. We could have a manifest with multiple importers pulling in data from each server into a single file, with each component represented as a separate node in the tree.

However, in many cases it will be simpler to have each component responsible for its own manifest, because the config is straightforward and less error-prone, and it's more resilient to certain components failing while others continue running.

We can imagine services auto-generating their own manifests from their monitor services and periodically sending those manifests to some central manager who uses the proposed new if-merge tool to collate them into a single file.

This would also make it much easier for multiple teams or organizations to work independently on their systems, but still interoperate and aggregate later, as they would all conform to the IF protocol rules.

pipelines, inputs and outputs from each node get added to the final manifest as tree components. Then there is a set of merging rules for the context.

The way this would be used is that you would receive N manifests and run if-merge ./folder-of-manifests then, in the resulting manifest add the aggregation config you want to execute and run again to produce aggregated values for the whole system. Perhaps later we can make the aggregation config injectable from the command line rather than having to open and modify the file.

merging rules

context:

• name provide the manifest name as an argument to if-merge
• tags: include all unique values from all component manifests
• description: can be provided as an argument to if-merge or can be a hardcoded default, e.g. merged manifest
• initialize: all the entries from all the component manifests should be added to the merged manifest intialize pipeline. If there are name-collisions, append the original manifest filename to the instance name (will also require the invocation in the node pipeline to get renamed too)

execution

• do not copy over execution information into the merged manifest. When the merged manifest is executed, it can generate its own execution information.

tree

• Each individual tree node from each manifest should become a tree node in the merged file. Any nesting present in the individual manifests should persist into the merged file.

Example

Given the following manifests:

name: manifest 1
description: >-
  a minimal manifest executing a single plugin on a single component for a
  single timestep
tags: null
initialize:
  plugins:
    sum:
      path: builtin
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy-sum
execution:
  command: >-
    /home/user/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/user/Code/if/src/index.ts -m manifests/examples/basic.yml -s
  environment:
    if-version: 0.4.0
    os: linux
    os-version: 5.15.0-107-generic
    node-version: 21.4.0
    date-time: 2024-06-19T10:50:25.720Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@grnsft/if-core@0.0.7'
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child-0:
      defaults:
        cpu/thermal-design-power: 100
      pipeline:
        - sum
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
        - timestamp: 2023-07-06T00:01
          duration: 1
          cpu/utilization: 80
          cpu/energy: 30
          network/energy: 30
        - timestamp: 2023-07-06T00:02
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
          cpu/thermal-design-power: 100
          energy-sum: 50
        - timestamp: 2023-07-06T00:01
          duration: 1
          cpu/utilization: 80
          cpu/energy: 30
          network/energy: 30
          cpu/thermal-design-power: 100
          energy-sum: 60
        - timestamp: 2023-07-06T00:02
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
          cpu/thermal-design-power: 100
          energy-sum: 50

name: manifest 2
description: >-
  a minimal manifest executing a single plugin on a single component for a
  single timestep
tags: null
initialize:
  plugins:
    multiply:
      path: builtin
      method: Multiply
      global-config:
        input-parameters:
          - cpu/utilization
          - duration
        output-parameter: cpu-times-duration
execution:
  command: >-
    /home/user/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/user/Code/if/src/index.ts -m manifests/examples/basic.yml -s
  environment:
    if-version: 0.4.0
    os: linux
    os-version: 5.15.0-107-generic
    node-version: 21.4.0
    date-time: 2024-06-19T10:51:42.054Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@grnsft/if-core@0.0.7'
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child-0:
      defaults:
        cpu/thermal-design-power: 100
      pipeline:
        - multiply
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
        - timestamp: 2023-07-06T00:01
          duration: 1
          cpu/utilization: 80
          cpu/energy: 30
          network/energy: 30
        - timestamp: 2023-07-06T00:02
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
          cpu/thermal-design-power: 100
          cpu-times-duration: 20
        - timestamp: 2023-07-06T00:01
          duration: 1
          cpu/utilization: 80
          cpu/energy: 30
          network/energy: 30
          cpu/thermal-design-power: 100
          cpu-times-duration: 80
        - timestamp: 2023-07-06T00:02
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
          cpu/thermal-design-power: 100
          cpu-times-duration: 20

If I run if-merge manifest1.yml manifest2.yml --name merged-manifest --description "description of my manifest" -o merged-manifest.yml

Then I get the following output file:

name: merged-manifest
description: description of my manifest
tags: null
initialize:
  plugins:
    multiply:
      path: builtin
      method: Multiply
      global-config:
        input-parameters:
          - cpu/utilization
          - duration
        output-parameter: cpu-times-duration
    sum:
      path: builtin
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy-sum
tree:
  children:
    child-0:
      defaults:
        cpu/thermal-design-power: 100
      pipeline:
        - sum
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
        - timestamp: 2023-07-06T00:01
          duration: 1
          cpu/utilization: 80
          cpu/energy: 30
          network/energy: 30
        - timestamp: 2023-07-06T00:02
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
          cpu/thermal-design-power: 100
          energy-sum: 50
        - timestamp: 2023-07-06T00:01
          duration: 1
          cpu/utilization: 80
          cpu/energy: 30
          network/energy: 30
          cpu/thermal-design-power: 100
          energy-sum: 60
        - timestamp: 2023-07-06T00:02
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
          cpu/thermal-design-power: 100
          energy-sum: 50
    child-0-manifest2:
      defaults:
        cpu/thermal-design-power: 100
      pipeline:
        - multiply
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
        - timestamp: 2023-07-06T00:01
          duration: 1
          cpu/utilization: 80
          cpu/energy: 30
          network/energy: 30
        - timestamp: 2023-07-06T00:02
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
          cpu/thermal-design-power: 100
          cpu-times-duration: 20
        - timestamp: 2023-07-06T00:01
          duration: 1
          cpu/utilization: 80
          cpu/energy: 30
          network/energy: 30
          cpu/thermal-design-power: 100
          cpu-times-duration: 80
        - timestamp: 2023-07-06T00:02
          duration: 1
          cpu/utilization: 20
          cpu/energy: 20
          network/energy: 30
          cpu/thermal-design-power: 100
          cpu-times-duration: 20

[jmcook1186]

@jawache I'd like to see this feature available in IF, ideally as part of i/o epic - wdyt?