search

Green-Software-Foundation / standards-wg

GSF Standards Working Group
Other
30 stars 4 forks source link

2024.08.29 #120

Open seanmcilroy29 opened 3 weeks ago

seanmcilroy29 commented 3 weeks ago

2024.08.29 Agenda/Minutes

Time 1600 (GMT) - See the time in your timezone

Antitrust Policy

Joint Development Foundation meetings may involve participation by industry competitors, and the Joint Development Foundation intends to conduct all of its activities in accordance with applicable antitrust and competition laws. It is, therefore, extremely important that attendees adhere to meeting agendas and be aware of and not participate in any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws.

If you have questions about these matters, please contact your company counsel or counsel to the Joint Development Foundation, DLA Piper.

Recordings

WG agreed to record all Meetings. This meeting recording will be available until the next scheduled meeting.

Roll Call

Please add 'Attended' to this issue during the meeting to denote attendance.

Any untracked attendees will be added by the GSF team below:

Agenda

Use Case submission

Discussion Continued - Diversifying and Developing the SCI Standard - Henry

Last Meeting AIs

Project Review updates

For Review

Note: WG use case template submission - After submitting this issue, your use case will be submitted to the WG Agenda for discussion. Article submission—Once you submit this issue, it will be assigned to the GSF Editor for review.

Future meeting Agenda submissions

Next Meeting

Adjourn

Standing Agenda / Future Agenda submissions

PindyBhullar commented 2 weeks ago

I have reached out to my friend from Ecolab to give us an overview of the water usage in data centres and the terminology, issues, opportunities associated with this resource. He has been away on holiday and I am waiting to hear back when he can present to this group. As soon as I have a date that fits his schedule I will let the chairs know.

PindyBhullar commented 2 weeks ago

TOSS update - I am working through the decision trees post the workshops. I will need a few more weeks to update them all, and then seek feedback from co-chair.

tholewis commented 2 weeks ago

🌳 Attended!

PindyBhullar commented 2 weeks ago

Attended only part of the meeting

filga commented 2 weeks ago

Attended

srini1978 commented 2 weeks ago

Attended

sasthana-tw commented 2 weeks ago

Attended

navveenb commented 2 weeks ago

Attended

Henry-WattTime commented 2 weeks ago

Attended

seanmcilroy29 commented 2 weeks ago

Attended

seanmcilroy29 commented 2 weeks ago

MoM - GSF Standards Working Group meeting

EcoDoc Sense Tool Overview The GSF Standards Working Group discussed the development of EcoDoc Sense. This tool evaluates software UCL, which introduces the problem of assessing environmental impact during the software design phase rather than the development phase. The tool EcoDoc Sense takes software architecture documents as input and generates a sustainability report. The UCL presenter explains the tool's home page, which includes links to research papers, API documentation, and the Green Software Foundation. The tool allows users to view sample results and analyze new documents, with a progress meter showing the analysis phases.

Tool Demonstration and Analysis Process The UCL presenter demonstrates uploading a Dropbox architecture document and the analysis process, including a timer and progress meter. The tool provides detailed queries and explanations for each practice to improve sustainability. The presenter also explains the time-consuming nature of the pipeline and the importance of the UI in starting it. The tool analyzes the document in three phases: pre-processing, retrieval, and generation, each of which has its own importance.

Retrieval Phase and Retriever Selection UCL presenter discusses the retrieval phase, which involves getting context embeddings and queries from the green practice database. Three types of retrievers are explored: Chroma with MMR algorithm, multiple retrievers, and ensemble retrievers. The performance of each retriever is evaluated, with Chroma MMR being chosen as the baseline due to its efficiency. The retrieval phase aims to retrieve relevant context and queries to create a contextualized prompt for the generator.

Generation Phase and Model Fine-Tuning UCL presenter explains the generation phase, where contextualized prompts are fed into the generator to generate responses. The model is fine-tuned using GPT-4 to improve performance, with a line chart showing the improvement from the baseline model. The tool tests different configurations of retrievers and generators to find the best combination. The final configuration chosen is Chroma MMR with a fine-tuned generator, which performs well in both streaming and MCC metrics.

Sustainability Report and Improvement Plan UCL presenter summarizes the workflow, including pre-processing, retrieval, and generation phases. The sustainability report includes a summary, table of contents, graphical evaluation, and improvement plan. The improvement plan identifies specific improvements needed for the software architecture document. The report provides detailed explanations for each green practice and the corresponding queries to evaluate the document.

Validation and Evaluation of LLM Results The UCL presenter explained the validation process, including the manual evaluation of documents to create a ground-truth data set. The LLM's responses are compared with the ground-truth data set to evaluate its performance. Metrics such as G mean, and MCC optimize true positives and assess the model's performance. Different models are explored, with five from Microsoft chosen for better performance and fewer hallucinations.

Trustwise's Optimized AI Solution Shiv introduces Trustwise's solution for making generative AI applications optimal, carbon neutral, safe, and efficient. The solution uses the SCI standard and focuses on reducing costs and energy savings while being compliant with it. The methodology includes initial assessment, optimization strategies, and running workloads through the solution. The solution helps enterprises pick the right mix of configurations for optimal carbon footprint, energy efficiency, and cost.

Case Study and Implementation Roadmap Shiv provides a case study of an insurance company automating its home insurance policy chatbot. The solution uses different models and configurations to optimize the workload and reduce carbon footprint and cost. The implementation roadmap includes defining the software boundary, functional unit, and quantification methods. The solution generates a leaderboard and report to help clients pick the optimal configuration for their use case.

Feedback and Next Steps Henry and Shiv discuss the table's comparability and suitability for decision-making. The table shows different metrics for each model, including SCI scores, hallucinations, and regulatory compliance. The solution provides a powerful tool for users to choose the correct configuration based on their needs. Henry and Shiv agree to work together to amplify the case study and connect with the communications team for further discussion.

Action Items

srini1978 commented 2 weeks ago

@seanmcilroy29 Can you please share the link to the recording ?

seanmcilroy29 commented 5 days ago

@srini1978 - Meeting recording