search

dotnet / dnceng

.NET Engineering Services
MIT License
25 stars 18 forks source link

node+npm needed on build images for Windows, and Mariner-2 #3091

Open radical opened 3 months ago

radical commented 3 months ago

dotnet/aspire CI is running on https://helix.dot.net/#heading-windows-vs2022preview-amd64-open and https://helix.dot.net/#heading-1es-mariner-2 and those don't appear to have node+npm installed on them by default. We need these for running some new tests.

Linux at least has node installed though, but no npm. And Windows is missing both.

cc @RussKie @joperezr

Release Note Category

radical commented 2 months ago

@dougbu any ETA on this one? We have tests blocked on dotnet/aspire due to this. cc @eerhardt

dougbu commented 2 months ago

Sorry. I haven't had a chance to look. I'll keep a tab open here and get back to you b/4 Monday

dougbu commented 2 months ago

oh, I mixed this up w/ #3122. this issue is part of a "spike" of operations issues we're working through. it's no longer really on our backlog, more a matter of finding an assignee and / or deciding what we do when

could you speak to reason behind the urgency @radical❓ for example, why are automated tests on additional platforms needed❓

/cc @ilyas1974 and @garath

radical commented 2 months ago

or example, why are automated tests on additional platforms needed❓

Using node+npm projects with dotnet/aspire is an explicit feature of the product, in parallel to all the others, and utilizes explicit aspire APIs for it. So, running node+npm dependent tests is not a different platform for us, it is just testing one part of the product.

Does that answer your question?

dougbu commented 2 months ago

or example, why are automated tests on additional platforms needed❓

Using node+npm projects with dotnet/aspire is an explicit feature of the product, in parallel to all the others, and utilizes explicit aspire APIs for it. So, running node+npm dependent tests is not a different platform for us, it is just testing one part of the product.

Does that answer your question?

not exactly. if you can test node+npm on Linux, what's the urgency behind also testing those scenarios on Windows❓

radical commented 2 months ago

cc @eerhardt

eerhardt commented 2 months ago

if you can test node+npm on Linux, what's the urgency behind also testing those scenarios on Windows❓

Because they are different, and we can have different bugs based on the OS. One example is here:

https://github.com/dotnet/aspire/pull/5142/files#diff-7a45254774650a016a455a5455a121b2ea58a5d9f5ee718dadd0a25127559967

npm isn't npm.exe on Windows, but it is npm.cmd. So when looking for the command the existing code wasn't finding it correctly.

radical commented 3 weeks ago

Any update on this? Also Linux image has node 12 installed which reached EOL two years ago. This is actively blocking testing on dotnet/aspire. https://github.com/dotnet/aspire/pull/2225#issuecomment-2348285240 cc @davidfowl

riarenas commented 4 days ago

I will start looking at this today.

A few questions.

https://helix.dot.net/#heading-1es-mariner-2

Is an image we get from 1ES, and not something we should modify, so we need to decide how we want to fix this scenario:

If neither of these options help, we will need to create a new image, which we can try to base off the existing mariner image, but it will take some experimentation to get that working end to end.

It seems we at least think that we install a portable version of Node 16 in the windows image under C:\arcade-tools We install node 10 and 16 in this way. Can you check if this installation would help with your needs?

image

radical commented 3 days ago

Can you use docker instead of needing NPM on the host?

This needs to be explored, and would be non-trivial to implement. Especially, because aspire depends on talking to docker, and creating/building/running images.

Alternatively, could you try moving to the build.ubuntu.* images from helix.dot.net#1ESHostedPoolImagesWestUS-rg-Internal-Linux where we do install our artifacts?

How different is this from the existing images?

Also, I want to confirm that I shared the correct information. These are the helix queues that we use specifically, where we want this to be available:

            HelixTargetQueues: Windows.11.Amd64.Client.Open
            HelixTargetQueues: Windows.11.Amd64.Client

            HelixTargetQueues: Ubuntu.2204.Amd64.Open
            HelixTargetQueues: Ubuntu.2204.Amd64
riarenas commented 3 days ago

How different is this from the existing images?

The software installed in the images is located in the Helix website. We have full control over the build.ubuntu images, while anything that starts with 1es-* is provided as is by 1ES and we don't control, or even know fully what is installed in them.

Also, I want to confirm that I shared the correct information. These are the helix queues that we use specifically, where we want this to be available:

That is certainly different from the issue description. I was looking at the build images, not the test images. Do you need this for build, or for tests in helix? (or both?)

radical commented 3 days ago

Also, I want to confirm that I shared the correct information. These are the helix queues that we use specifically, where we want this to be available:

That is certainly different from the issue description. I was looking at the build images, not the test images. Do you need this for build, or for tests in helix? (or both?)

Oh my apologies. Yes, the initial ask was on build images because most of our tests were running there. But now we have moved most to helix. So, we need this on the helix images too.

riarenas commented 3 days ago

Thanks for the clarification! I can't find us installing node in the windows helix queues, and looks like we only install really old node versions in the ubuntu queue.

So to clarify, the places where node + npm are needed:

I'm making the following assumptions:

Let me know if either of these assumptions don't hold!

radical commented 2 days ago

Thanks for the clarification! I can't find us installing node in the windows helix queues, and looks like we only install really old node versions in the ubuntu queue.

So to clarify, the places where node + npm are needed:

  • Build images:

    • windows.vs2022preview.amd64[.open]
    • build.ubuntu.2204.amd64[.open]

  • Helix test queues:

    • windows.11.amd64.client[.open]
    • ubuntu.2204.amd64[.open]

Correct.

I'm making the following assumptions:

  • The latest LTS (node 20) would work

Yes.

  • You are able to use the build.ubuntu.* ubuntu build images provided by dnceng, as we shouldn't install additional software directly over the 1es-mariner-2 image.

Yes, I will try to move us to that.

Thank you!