search

Green-Software-Foundation / carbon-aware-sdk

Carbon-Aware SDK
https://carbon-aware-sdk.greensoftware.foundation/
MIT License
456 stars 98 forks source link

.NET8 blog article #501

Closed Sophietn closed 1 week ago

Sophietn commented 4 months ago

What: A blog article outlining everything the Carbon Aware SDK team did for the .NET8 upgrade to the Carbon Aware SDK toolkit. Written by NTT DATA (who largely completed the work needed for the .NET8 upgrade) Why: A celebration of the archivement, useful resource for others wanting to upgrade, recognises NTT DATA's contributions to the CA SDK and their adoption. SoW:

YaSuenag commented 4 months ago

.NET 8 upgrade is a significant work by @tiwatsuka from NTT .
So I'd like to contribute an article with him. @nttDamien from NTT DATA will support our work.

Define what the article should contain

@Sophietn How about the following? Please give me your feedback.

  1. Why we need .NET 8?
  2. What it takes to upgrade to .NET 8
  3. Unexpected work items
    • Testcase update for Azure Functions
    • Breaking changes about port number of ASP.NET
    • Fix build pipeline on GitHub Actions
  4. Use cases in NTT / NTT DATA after .NET 8 upgrade
Sophietn commented 4 months ago

Thank you! @YaSuenag This looks like a good structure to me. Looking forward to reviewing the article you draft.

YaSuenag commented 3 months ago

@Sophietn @vaughanknight @danuw I share the article of .NET8 migration work on CASDK. Can you review? This article is written by @tiwatsuka , @nttDamien , and me.

Carbon Aware SDK 1.4, behind the scenes

As most software nowadays, the Carbon Aware SDK relies on a stack of utilities, and while adding a new feature is usually the most appealing for a project, it’s also critical to maintain the stack, especially in a community effort.

Containerization has helped shifting the upgrading work to a more convenient time for the development team, but there are still various motivation for keeping a stack up to date with current versions: security, bug fixes, performance, support… but the best it to couple with new feature development: such was the case for .NET framework.

However, those updates often have ripple effects, as their dependencies are not always foreseeable, making software upgrade workload hard to predict.

As NTT and NTT DATA were key participants in this contribution, this is a good occasion to cast a light on the behind the scenes, and the way this new Carbon Aware SDK is being used internally.

Why .NET 8 ?

Carbon Aware SDK v1.4.0 was released on May 2024, its core evolution was the upgrade to .NET 8. Until v1.3.x, the Carbon Aware SDK has relied on the LTS (Long Term Support) version .NET 6. With an EOL (End of Life) set for November 2024, an upgrade was unavoidable.

Microsoft released .NET 8 in Nov 2023, this is the latest LTS version of .NET and will be supported until Nov 2026.

Wanting to display the Carbon Intensity metrics from the Carbon Aware SDK WebAPI, made .NET 8 a requirement, as .NET 8 introduced an enhanced supports for implementing metrics features.

Indeed, the newly introduced IMeterFactory interface enabled us to create a Meter instance while maintaining modularity by using dependency injection (i.e. use the .NET 8 implementation of the feature, instead of recreating it… another software development sustainable pattern).

In summary, Carbon Intensity metrics handling was combined with the necessary support extension that .NET 8 upgrade provides.

In practice

The initial work for upgrading to .NET 8 was done in Pull Request #404 (aka PR, basically a code change proposal, which once approved will be merged in the main code).

Without being a C# expert, it’s still interesting to look at the PR and see that: it involves several individuals working together as a community, many files were impacted, tests and samples are as critical as they should.

For the nitty gritty (else jump to the next paragraph): the core work is “simply” updating the target framework version.

It can be done in the property window of each C# projects, for example in the Japanese version of Visual Studio (Fig.1).

fig1

Fig.1 Property window of C# project in Carbon Aware SDK on Visual Studio Community Edition

Carbon Aware SDK includes 30 C# projects (in v1.3.0 at least), so automation is welcomed. The target framework version is described in /Project/PropertyGroup/TargetFramework in .csproj file. For example, running the command on WSL:

find . -name "*.csproj" -exec sed -i 's|^\(\s\+\)<TargetFramework>net6.0</TargetFramework>$|\1<TargetFramework>net8.0</TargetFramework>|g' {} \;

.NET version is specified in many other places, which need to be updated as well (grep will list them all).

While the updating part is done, the work does not end there…

Unexpected work items

While the .NET 8 upgrade was done, some unexpected issues surfaced.

Ripple effect on sample code

To help onboard new comers to the Carbon Aware SDK, a sample running on Azure Functions is provided.

Azure Functions for .NET is transitioning one of its execution modes (the In-process model) for the Isolated worker model (more details here). Moreover, .NET 8 did not provide yet an option to use the former model in its initial release (cf. roadmap of Azure Functions).

As our sample was still implementing the in-process model (to be deprecated and not available in .NET 8 at this time), it made sense to migrate to the isolated worker model.

For the code lover, there is an helpful guide for the migration. This led to:

For more details browse: Pull Request #420.

Port Number Breaking change

As Carbon Aware SDK WebAPI uses ASP.NET Core technology another collateral must do change was required since .NET 8 changed its default port number from 80 to 8080 Microsoft Learn document).

Changing the port number from WebAPI container, which affects the containerPort in Helm chart and some GitHub Workflows which uses WebAPI.

Broken build pipeline on GitHub Actions

Thanks to GitHub, a lot of automation is available in order to publish code, allowing to focus more on coding, in particular the Carbon Aware SDK repository is configured to publish WebAPI container image (like a snapshot build) when a commit occurs on the dev branch.

However, it suddenly stopped working after .NET 8 upgrade.

The team investigated the logs (Fig. 2), as a container image for both AMD64 and Arm64 Linux in GitHub Actions with docker/build-push-action: a mysterious segmentation fault (SEGV) was occurring after the upgrade… the code was not changed, dotnet publish was outside the scope.

 > [linux/arm64 build-env 4/6] RUN dotnet publish CarbonAware.WebApi/src/CarbonAware.WebApi.csproj -c Release -o publish:
7.209 MSBuild version 17.9.6+a4ecab324 for .NET
24.69   Determining projects to restore...
41.42 Segmentation fault (core dumped)

Fig.2 Logs in dotnet publish on GitHub Actions

Further investigation was done, and thanks to a .NET blog, about multi-platform container support, that an unsupported approach was used for the build, and needed to be amended. More precisely, since .NET 6, QEMU static binaries were used to build container image for multi platforms.

Fortunately .NET blog guides how to build multi platform container images, and the workflow was fixed accordingly in Pull Request #498. So WebAPI container image with .NET 8 can be pulled from GitHub Packages now!

Use case in NTT / NTT DATA

While NTT & NTT DATA have been contributing to the Carbon Aware SDK a long time, none appeared in the adopters list, it is now changed thanks to those uses cases.

Carbon Intensity map

Thanks to the new Carbon Aware SDK v1.4.0 carbon metrics exporter (thanks to .NET 8), more visualization capability are reachable.

This feature facilitate integration with monitoring solutions like Prometheus and furthermore with a visualization solution like Grafana: unlocking geomap style visualization (showing metrics at specified locations on a map). By enabling the exporter and making some settings on Grafana, carbon intensities can be exported from Carbon Aware SDK to a geomap, this is part of a dashboard to monitor carbon emissions for software systems.

fig3

Green Dashboard for Kubernetes

Carbon Aware SDK helps increase awareness around Carbon emission, and it is now possible to monitor carbon emission per application within Kubernetes.

In practice, each container energy consumption is evaluated through Kepler (sandbox project in Cloud Native Cloud Foundation, CNCF), and thanks to the Carbon Aware SDK, the carbon emission can be evaluated.

All those emission data from power grid can be accessed through Prometheus exporter with Carbon Aware SDK (starting v1.4.0), and a visualized with Grafana dashboard.

The power consumption, energy consumption, carbon emission, and SCI (Software Carbon Intensity) can be seen at a glance!

fig4

danuw commented 3 months ago

awesome @YaSuenag and team - can we also get that into the blog please? https://github.com/Green-Software-Foundation/carbon-aware-sdk/tree/dev/casdk-docs/blog

will review later and come with feedback if I have any

NAMRATA-WOKE commented 4 weeks ago

@YaSuenag @tiwatsuka @nttDamien, thank you for sharing this article! It was a bit challenging to edit the document on GitHub so I created a Google doc. Please see the updated version here: https://docs.google.com/document/d/1bM7xCDj3tXDQL-TYCyX8hsdsnSLtRbTIDD4qF0Q1mf0/edit?usp=sharing

I have some questions in the comments for you. Since this is in some ways also a use case on NTT DATA, I'd like to share this with the Opensource Working Group for their feedback and to see if there are any questions they'd like this article to address.

@danuw, if you can provide your feedback in the Google doc, that would be great!

nttDamien commented 3 weeks ago

Hi,

Thanks a lot for your review and comments. WIth Suenaga san, we posted some reply to the comments.

Looking forward, Damien

From: Namrata @.> Sent: Wednesday, July 31, 2024 3:16 To: Green-Software-Foundation/carbon-aware-sdk @.> Cc: TIG ＲＯＵＸ Ｄａｍｉｅｎ/Roux, Damien (NTT DATA) @.>; Mention @.> Subject: Re: [Green-Software-Foundation/carbon-aware-sdk] .NET8 blog article (Issue #501)

@YaSuenaghttps://urldefense.com/v3/__https://github.com/YaSuenag__;!!GCTRfqYYOYGmgK_z!69Y3zzQlXfPGhkdBZP02QodeBZIQbZmjH0oI67rM2PPx236xl7DmwavIYW3uMPwXuTacqqVS9nZ42Vl9dd1nwi-7Bnrcdw$ @tiwatsukahttps://urldefense.com/v3/__https://github.com/tiwatsuka__;!!GCTRfqYYOYGmgK_z!69Y3zzQlXfPGhkdBZP02QodeBZIQbZmjH0oI67rM2PPx236xl7DmwavIYW3uMPwXuTacqqVS9nZ42Vl9dd1nwi-P9PQ--Q$ @nttDamienhttps://urldefense.com/v3/__https://github.com/nttDamien__;!!GCTRfqYYOYGmgK_z!69Y3zzQlXfPGhkdBZP02QodeBZIQbZmjH0oI67rM2PPx236xl7DmwavIYW3uMPwXuTacqqVS9nZ42Vl9dd1nwi95JK8EJw$, thank you for sharing this article! It was a bit challenging to edit the document on GitHub so I created a Google doc. Please see the updated version here: https://docs.google.com/document/d/1bM7xCDj3tXDQL-TYCyX8hsdsnSLtRbTIDD4qF0Q1mf0/edit?usp=sharinghttps://urldefense.com/v3/__https://docs.google.com/document/d/1bM7xCDj3tXDQL-TYCyX8hsdsnSLtRbTIDD4qF0Q1mf0/edit?usp=sharing__;!!GCTRfqYYOYGmgK_z!69Y3zzQlXfPGhkdBZP02QodeBZIQbZmjH0oI67rM2PPx236xl7DmwavIYW3uMPwXuTacqqVS9nZ42Vl9dd1nwi8pNY-3Xg$

I have some questions in the comments for you. Since this is in some ways also a use case on NTT DATA, I'd like to share this with the Opensource Working Group for their feedback and to see if there are any questions they'd like this article to address.

@danuwhttps://urldefense.com/v3/__https://github.com/danuw__;!!GCTRfqYYOYGmgK_z!69Y3zzQlXfPGhkdBZP02QodeBZIQbZmjH0oI67rM2PPx236xl7DmwavIYW3uMPwXuTacqqVS9nZ42Vl9dd1nwi9beyDTVA$, if you can provide your feedback in the Google doc, that would be great!

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/Green-Software-Foundation/carbon-aware-sdk/issues/501*issuecomment-2258934815__;Iw!!GCTRfqYYOYGmgK_z!69Y3zzQlXfPGhkdBZP02QodeBZIQbZmjH0oI67rM2PPx236xl7DmwavIYW3uMPwXuTacqqVS9nZ42Vl9dd1nwi-GARutOg$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A55V7R6VDJCNZSFU6XSQHTDZO7KANAVCNFSM6AAAAABGJCOBUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJYHEZTIOBRGU__;!!GCTRfqYYOYGmgK_z!69Y3zzQlXfPGhkdBZP02QodeBZIQbZmjH0oI67rM2PPx236xl7DmwavIYW3uMPwXuTacqqVS9nZ42Vl9dd1nwi-ZkUOmvw$. You are receiving this because you were mentioned.Message ID: @.***>

NAMRATA-WOKE commented 3 weeks ago

Thanks for responding to the comments @nttDamien. It might be useful to get feedback from others subscribed to the CASDK project and the PLs and Working Group Chairs. But I wanted to check to see if that's already happened?

cc. @seanmcilroy29 @russelltrow

nttDamien commented 3 weeks ago

[like] TIG ＲＯＵＸ Ｄａｍｉｅｎ/Roux, Damien (... reacted to your message:

From: Namrata @.> Sent: Thursday, August 1, 2024 12:59:00 AM To: Green-Software-Foundation/carbon-aware-sdk @.> Cc: TIG ＲＯＵＸ Ｄａｍｉｅｎ/Roux, Damien (NTT DATA) @.>; Mention @.> Subject: Re: [Green-Software-Foundation/carbon-aware-sdk] .NET8 blog article (Issue #501)

Thanks for responding to the comments @nttDamienhttps://urldefense.com/v3/__https://github.com/nttDamien__;!!GCTRfqYYOYGmgK_z!7mrPNzImKiAkJbeUiZkg6RcbeBSkSoebQ7FD2eydZIPjOFH6cAoKqOZ2DxarrpaOVeXNkJxWVw2-QjWDDJCq-Vv9LWvlbw$. It might be useful to get feedback from others subscribed to the CASDK project and the PLs and Working Group Chairs. But I wanted to check to see if that's already happened?

cc. @seanmcilroy29https://urldefense.com/v3/__https://github.com/seanmcilroy29__;!!GCTRfqYYOYGmgK_z!7mrPNzImKiAkJbeUiZkg6RcbeBSkSoebQ7FD2eydZIPjOFH6cAoKqOZ2DxarrpaOVeXNkJxWVw2-QjWDDJCq-Vslo41s1Q$ @russelltrowhttps://urldefense.com/v3/__https://github.com/russelltrow__;!!GCTRfqYYOYGmgK_z!7mrPNzImKiAkJbeUiZkg6RcbeBSkSoebQ7FD2eydZIPjOFH6cAoKqOZ2DxarrpaOVeXNkJxWVw2-QjWDDJCq-VsUR952Xw$

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/Green-Software-Foundation/carbon-aware-sdk/issues/501*issuecomment-2261738608__;Iw!!GCTRfqYYOYGmgK_z!7mrPNzImKiAkJbeUiZkg6RcbeBSkSoebQ7FD2eydZIPjOFH6cAoKqOZ2DxarrpaOVeXNkJxWVw2-QjWDDJCq-VugRXvRFw$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A55V7R4USZO2NNF7U3AX3JLZPGB5JAVCNFSM6AAAAABGJCOBUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRRG4ZTQNRQHA__;!!GCTRfqYYOYGmgK_z!7mrPNzImKiAkJbeUiZkg6RcbeBSkSoebQ7FD2eydZIPjOFH6cAoKqOZ2DxarrpaOVeXNkJxWVw2-QjWDDJCq-VuXR--GiQ$. You are receiving this because you were mentioned.Message ID: @.***>

NAMRATA-WOKE commented 3 weeks ago

@danuw have you had a chance to review and offer any feedback? https://docs.google.com/document/d/1bM7xCDj3tXDQL-TYCyX8hsdsnSLtRbTIDD4qF0Q1mf0/edit?usp=sharing

danuw commented 3 weeks ago

Done - sorry for delay - Great and only saw a possible typo in the last sentence which i flagged through a comment.

Thanks.

NAMRATA-WOKE commented 3 weeks ago

@Jenya-design can you create an image for this article? Use the Google Doc as a reference.

Jenya-design commented 2 weeks ago

@NAMRATA-WOKE please check the cover attached and on Drive Image

NAMRATA-WOKE commented 2 weeks ago

Article is scheduled to be published Tuesday next week (Aug 13). Thanks everyone!

NAMRATA-WOKE commented 2 weeks ago

Hi all, please note the article has been published. Let me know if any changes are needed: https://greensoftware.foundation/articles/upgrading-to-net-8-inside-the-carbon-aware-sdk-v1-4

nttDamien commented 2 weeks ago

[celebrate] TIG ＲＯＵＸ Ｄａｍｉｅｎ/Roux, Damien (... reacted to your message:

From: Namrata @.> Sent: Tuesday, August 13, 2024 12:08:52 PM To: Green-Software-Foundation/carbon-aware-sdk @.> Cc: TIG ＲＯＵＸ Ｄａｍｉｅｎ/Roux, Damien (NTT DATA) @.>; Mention @.> Subject: Re: [Green-Software-Foundation/carbon-aware-sdk] .NET8 blog article (Issue #501)

Hi all, please note the article has been published. Let me know if any changes are needed: https://greensoftware.foundation/articles/upgrading-to-net-8-inside-the-carbon-aware-sdk-v1-4https://urldefense.com/v3/__https://greensoftware.foundation/articles/upgrading-to-net-8-inside-the-carbon-aware-sdk-v1-4__;!!GCTRfqYYOYGmgK_z!5uWI9585_X1euQXunStLg0CE6NFSVVDsZO2yyGUPSsNiA9EOPP1ny_7EGrWYnnGfOxsMWfgUIyEvBD9B_KbunsJTyeqfwg$

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/Green-Software-Foundation/carbon-aware-sdk/issues/501*issuecomment-2286092768__;Iw!!GCTRfqYYOYGmgK_z!5uWI9585_X1euQXunStLg0CE6NFSVVDsZO2yyGUPSsNiA9EOPP1ny_7EGrWYnnGfOxsMWfgUIyEvBD9B_KbunsLPWqWkDA$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A55V7R2FEAQCQS5RFTPRBWTZRHZNJAVCNFSM6AAAAABGJCOBUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBWGA4TENZWHA__;!!GCTRfqYYOYGmgK_z!5uWI9585_X1euQXunStLg0CE6NFSVVDsZO2yyGUPSsNiA9EOPP1ny_7EGrWYnnGfOxsMWfgUIyEvBD9B_KbunsL0jVxl0A$. You are receiving this because you were mentioned.Message ID: @.***>

nttDamien commented 2 weeks ago

Such a great news, thanks a lot !

From: Namrata @.> Sent: Tuesday, August 13, 2024 21:08 To: Green-Software-Foundation/carbon-aware-sdk @.> Cc: TIG ＲＯＵＸ Ｄａｍｉｅｎ/Roux, Damien (NTT DATA) @.>; Mention @.> Subject: Re: [Green-Software-Foundation/carbon-aware-sdk] .NET8 blog article (Issue #501)

Hi all, please note the article has been published. Let me know if any changes are needed: https://greensoftware.foundation/articles/upgrading-to-net-8-inside-the-carbon-aware-sdk-v1-4https://urldefense.com/v3/__https://greensoftware.foundation/articles/upgrading-to-net-8-inside-the-carbon-aware-sdk-v1-4__;!!GCTRfqYYOYGmgK_z!5uWI9585_X1euQXunStLg0CE6NFSVVDsZO2yyGUPSsNiA9EOPP1ny_7EGrWYnnGfOxsMWfgUIyEvBD9B_KbunsJTyeqfwg$

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/Green-Software-Foundation/carbon-aware-sdk/issues/501*issuecomment-2286092768__;Iw!!GCTRfqYYOYGmgK_z!5uWI9585_X1euQXunStLg0CE6NFSVVDsZO2yyGUPSsNiA9EOPP1ny_7EGrWYnnGfOxsMWfgUIyEvBD9B_KbunsLPWqWkDA$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A55V7R2FEAQCQS5RFTPRBWTZRHZNJAVCNFSM6AAAAABGJCOBUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBWGA4TENZWHA__;!!GCTRfqYYOYGmgK_z!5uWI9585_X1euQXunStLg0CE6NFSVVDsZO2yyGUPSsNiA9EOPP1ny_7EGrWYnnGfOxsMWfgUIyEvBD9B_KbunsL0jVxl0A$. You are receiving this because you were mentioned.Message ID: @.***>

YaSuenag commented 2 weeks ago

Thanks a lot @NAMRATA-WOKE !!

BTW why screenshot of our dashboards in ".NET8 Upgrade in Action" have not posted?

NAMRATA-WOKE commented 2 weeks ago

@YaSuenag Thanks for bringing that to my attention. The images should appear now!

danuw commented 1 week ago

.net 8 blog post published at https://carbon-aware-sdk.greensoftware.foundation/blog/dotnet-8-upgrade and https://greensoftware.foundation/articles/upgrading-to-net-8-inside-the-carbon-aware-sdk-v1-4