`azure-importer` memory outputs cannot used with `e-mem` plug-in

[pangteckchun]

Hi, while joining azure-importer outputs to e-mem plug-in, i get the following error:

[2024-04-22 04:44:44.991 PM] error:     "memory/capacity" parameter is required. Error code: invalid_type.
InputValidationError:   "memory/capacity" parameter is required. Error code: invalid_type.
    at validate (C:\Users\p1337447\AppData\Roaming\npm\node_modules\@grnsft\if-plugins\build\util\validations.js:49:15)
    at validateSingleInput (C:\Users\p1337447\AppData\Roaming\npm\node_modules\@grnsft\if-plugins\build\lib\e-mem\index.js:55:43)
    at C:\Users\p1337447\AppData\Roaming\npm\node_modules\@grnsft\if-plugins\build\lib\e-mem\index.js:16:31
    at Array.map (<anonymous>)
    at Object.execute (C:\Users\p1337447\AppData\Roaming\npm\node_modules\@grnsft\if-plugins\build\lib\e-mem\index.js:15:23)
    at computeNode (C:\Users\p1337447\AppData\Roaming\npm\node_modules\@grnsft\if\build\lib\compute.js:57:36)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async traverse (C:\Users\p1337447\AppData\Roaming\npm\node_modules\@grnsft\if\build\lib\compute.js:11:9)
    at async compute (C:\Users\p1337447\AppData\Roaming\npm\node_modules\@grnsft\if\build\lib\compute.js:79:5)
    at async impactEngine (C:\Users\p1337447\AppData\Roaming\npm\node_modules\@grnsft\if\build\index.js:26:30)

We observed the issues with the outputs from azure-importer:

1. They are named differently than what's expected for downstream module consumption, e.g. e-mem plug-in

   • They should named without the "/GB" suffix; meaning they should be: memory/available, memory/used & memory/capacity instead.

2. The type for memory/capacity should be number not string.

Thank you for your attention and hope to hear from you soon!

TC

[jmcook1186]

hi @pangteckchun , thanks for raising this, I hadn't noticed this issue before. There are a few ways to correct this - the best way is to use the Coefficient plugin instead of EMem - we are going to deprecate and remove EMem soon anyway because the same outcome can easily be achieved using Coefficient - you simply provide the name of the field representing the memory used in GB and the coefficient you want to use for energy consumed by memory in kW/GB.

[pangteckchun]

Hi @jmcook1186 , thank you for the prompt reply. Appreciate this! I tried coefficient plug-in and it helps in my case but not fully. Let me explain why.

This is because it is a pure mathematical manipulation and it does not derive the memory/energy output for me which I need e-mem for. So I still need to use e-mem in my pipeline while I simply used coefficient to workaround the name of the parameter issue I highlighted above.

Hence the 2 issues highlighted remains but I was able to workaround it using coefficient to rename the parameter name and value changed to a number. But as mentioned, I need a plug-in to help derive the memory/energy output and e-mem does that as its very core; hence still very much needed. I will use the output from e-mem for downstream sci-e input to derive the total embodied carbon.

Please don't deprecate just yet until there is an alternative plug-in :-)

[jmcook1186]

I'm not sure I follow - you can name your output parameter memory/energy.

The energy-per-gb parameter is just a constant coefficient - we inherited it from the cloud carbon footprint version here https://www.cloudcarbonfootprint.org/docs/methodology#memory

The rationale for moving to Coefficient is to encourage people to be a little more critical of the value they choose to use for the kwH/GB parameter - you can simply choose to set coefficient to 0.000392 to reproduce the same behaviour as EMem and CCF, or you can use your own measured values or values from some other source if you prefer.

So, the following implementation of Coefficient is precisely equivalent to using EMem (you could even name this instance of Coefficient "EMem" if you want:

name: coefficient-demo
description:
tags:
initialize:
  outputs:
    - yaml
  plugins:
    emem-using-coefficient:
      method: Coefficient
      path: '@grnsft/if-plugins'
      global-config:
        input-parameter: 'memory/utilization'
        coefficient: 0.000392
        output-parameter: 'memory/energy'
tree:
  children:
    child:
      pipeline:
        - emem-using-coefficient
      config:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          memory/utilization: 30

[pangteckchun]

Hi Joseph,

I'm not sure I follow - you can name your output parameter memory/energy.

Yes, that's what I did to get the pipeline going. Thanks for the initial pointer. Was a savior for my day!

I understood the rationale now that you explained it - thank you.

Nice suggestion. And I adopted what you suggested to derive the final outcome i needed without using e-mem. But sticking to the formula e-mem used to calculate memory/energy, I needed 3 parameters to make the final output. Formula from e-mem GitHub page:

memory/energy = (('memory/utilization'/100) * 'memory/capacity') * energy-per-gb

This is config I did and it is working:

  "memory-utilization-actuals": # Custom - to calculate actual memory utilization from % given
      method: Divide
      path: "@grnsft/if-plugins"
      global-config:
        numerator: memory/utilization
        denominator: 100
        output: gcf-memory/utilization/actual
    "memory-used": # Custom - to calculate memory used in GB
      method: Multiply
      path: "@grnsft/if-plugins"
      global-config:
        input-parameters:
          ["gcf-memory/utilization/actual", "memory/capacity/GB"]
        output-parameter: "gcf-memory/used/GB"
    "emem-using-coefficient": # Custom - to calculate energy consumed by memory in kWh but not using e-mem plug-in
      method: Coefficient
      path: "@grnsft/if-plugins"
      global-config:
        input-parameter: "gcf-memory/used/GB"
        coefficient: 0.000392 # energy-per-gb Ref: https://www.cloudcarbonfootprint.org/docs/methodology/#memory
        output-parameter: "memory/energy" # retain IF parameter naming for downstream plug-ins

• memory/utilization is a percentage and needs to be divided by 100 to begin with
• Need to get memory used in GB before using coefficient
• Finally the last part is following your suggestion of using coefficient instead of e-mem.

Do let me know if you have a more elegant suggestion.

Otherwise this is no longer an issue.

[jmcook1186]

ok seems good. Also, if wrapping these calculations into a single plugin is easier for you, you can always keep your own fork of EMem and install it from your own Github or your local filesystem even if it gets deprecated and removed from our library - our whole ethos is about promoting the growth of an ecosystem of plugins to develop in repositories and npm packages that we don't maintain!

I'll close this for now but feel free to reach out again as you need.