Dear Samuel (@samuelrince) and Adrien (@adrienbanse),
Thank you for submitting your work to the Journal of Open Source Software. I enjoyed trying out EcoLogits and reading the manuscript.
Summary of submission
EcoLogits is a software package that transparently instruments popular LLM inference API clients so that they provide estimations for various metrics related to environmental impact, including energy consumption and operational and embodied carbon emission.
Installation and functionality were validated for the OpenAI provider.
Checklist items
I was able to easily check off all items on the JOSS review checklist except for Documentation > Functionality documentation.
The core functionality of EcoLogits is estimation, and estimations ought to be accompanied by a quantification of error/accuracy as part of its methodology. More specifically, I believe answers to questions including but not limited to the following will be very helpful, if not crucially important, to many users:
What are the sources of inaccuracies in estimation/modeling, and how much error (relative percentages or absolute quantities) can they introduce?
What is the implication of such inaccuracies in comparing estimates produced by different estimation methodologies? For instance, operational vs. embodied carbon emissions, or OpenAI vs. Anthropic?
When trying out the example code in the front page, I saw that some metrics show min and max values instead of a single estimate. Are they the actual minimum-possible and maximum-possible values with a guarantee that the real value will not be outside the range? If not, are they lower and upper bounds for a confidence interval?
This can be part of the Methodology page in the documentation. For those pertaining to the Impacts class (fields like min and max), additionally the API reference page would be appropriate. For the paper, given that space is limited, a brief summary and a few common case error quantifications (e.g., Chat completion via OpenAI) in the Methodology section would suffice.
Miscellaneous notes
All of the following are optional comments/requests/suggestions that I do not see as blocking or required.
Software
EcoLogits has excellent typing, but by monkeypatching the impacts instance attribute into API client response objects (e.g., ChatCompletion), it makes type checkers like Mypy and Pyright complain about missing attributes. I am very much aware that dynamic monkeypatching and static type checkers do not work well with each other, but in any case I think it would be nice to investigate. If this is infeasible, noting somewhere in the documentation about workarounds (e.g., typing.cast to ecologits.tracers.openai_tracer.ChatCompletion) could be helpful.
Paper
Title: I think "EcoLogits: Evaluating the Environmental Impacts of Generative AI" is more conventional.
Statement of Need, first paragraph: I think it is important to note that Meta's carbon emission claims are estimations.
Methodology, last paragraph: In "[...] we assess the environmental impacts of all components [...]," it seems like not all components are actually considered.
Reference: This is my fault, but currently, the Zeus software (https://ml.energy/zeus) and the Zeus academic paper (in NSDI) are only loosely connected. I would very much appreciate it if, in addition to the paper, you could cite Zeus with either its homepage (https://ml.energy/zeus) or GitHub repo (https://github.com/ml-energy/zeus).
Dear Samuel (@samuelrince) and Adrien (@adrienbanse),
Thank you for submitting your work to the Journal of Open Source Software. I enjoyed trying out EcoLogits and reading the manuscript.
Summary of submission
EcoLogits is a software package that transparently instruments popular LLM inference API clients so that they provide estimations for various metrics related to environmental impact, including energy consumption and operational and embodied carbon emission.
Installation and functionality were validated for the OpenAI provider.
Checklist items
I was able to easily check off all items on the JOSS review checklist except for Documentation > Functionality documentation.
The core functionality of EcoLogits is estimation, and estimations ought to be accompanied by a quantification of error/accuracy as part of its methodology. More specifically, I believe answers to questions including but not limited to the following will be very helpful, if not crucially important, to many users:
minandmaxvalues instead of a single estimate. Are they the actual minimum-possible and maximum-possible values with a guarantee that the real value will not be outside the range? If not, are they lower and upper bounds for a confidence interval?This can be part of the Methodology page in the documentation. For those pertaining to the
Impactsclass (fields likeminandmax), additionally the API reference page would be appropriate. For the paper, given that space is limited, a brief summary and a few common case error quantifications (e.g., Chat completion via OpenAI) in the Methodology section would suffice.Miscellaneous notes
All of the following are optional comments/requests/suggestions that I do not see as blocking or required.
Software
impactsinstance attribute into API client response objects (e.g.,ChatCompletion), it makes type checkers like Mypy and Pyright complain about missing attributes. I am very much aware that dynamic monkeypatching and static type checkers do not work well with each other, but in any case I think it would be nice to investigate. If this is infeasible, noting somewhere in the documentation about workarounds (e.g.,typing.casttoecologits.tracers.openai_tracer.ChatCompletion) could be helpful.Paper
Best regards, Jae-Won Chung
Review thread: https://github.com/openjournals/joss-reviews/issues/7471