search

dart-lang / test

A library for writing unit tests in Dart.
https://pub.dev/packages/test
495 stars 213 forks source link

empty coverage for wasm on chrome #2278

Open konsultaner opened 1 month ago

konsultaner commented 1 month ago

In my project https://github.com/konsultaner/connectanum-dart I use a vm and a wasm ci chain to produce a coverage output. I started to split up the two coverage outputs and found out that the wasm output is empty.

{"type":"CodeCoverage","coverage":[]}

I use this command to start the wasm coverage:

dart run test -p chrome --coverage=./coverage_chrome --timeout=5m --concurrency=1

I could reproduce it im my github action and locally on my linux system:

Dart SDK version: 3.5.2 (stable) (Wed Aug 28 10:01:20 2024 +0000) on "linux_x64"
Flutter 3.24.2 • channel stable • https://github.com/flutter/flutter.git
Framework • revision 4cf269e36d (vor 7 Tagen) • 2024-09-03 14:30:00 -0700
Engine • revision a6bd3f1de1
Tools • Dart 3.5.2 • DevTools 2.37.2
konsultaner commented 1 month ago

by the way, the test preformance difference without coverage and with coverage for wasm is 4x

jakemac53 commented 1 month ago

I don't think this is expected to work at this time, I am not sure we have the information that would be needed to collect coverage on wasm. cc @kevmoo do you know who to ask about this?

dart run test -p chrome --coverage=./coverage_chrome --timeout=5m --concurrency=1

Are you separately configuring wasm as the compiler in your dart_test.yaml? I just want to make sure this is actually a wasm thing and not a dart2js thing, that looks like a dart2js test.

by the way, the test preformance difference without coverage and with coverage for wasm is 4x

It definitely is expected to take longer, how much longer I don't know, but essentially every piece of code that is executed has to be tracked etc, coverage is very expensive to collect.

konsultaner commented 1 month ago 
platforms: [vm, chrome]
compilers: [dart2wasm]
on_platform:
  chrome: {timeout: 10x}

This really is a dart2wasm problem.

jakemac53 commented 1 month ago

Open questions/issues:

kevmoo commented 1 month ago

We do produce source maps. Need to chat with @osa1 and @mkustermann for details.

osa1 commented 1 month ago

Does chrome support coverage for wasm? They didn't previously.

I'm not sure. I've asked this to the relevant team in chat. I'll update. I've confirmed manually that Chrome currently generates coverage for Wasm. Correction: it seems like the answer is no, the coverage info I was seeing were all .js files. Chromium issue: https://g-issues.chromium.org/issues/354020953.

Does dart2wasm produce source maps?

Yes.

We probably need support from source_map_stack_trace

I'm working on implementing a Wasm stack trace mapper in test. If needed I'll update source_map_stack_trace as well.

How much work will it be to plumb all of those together? Hopefully we can borrow/share code with the JS coverage support

I don't know the details of test package, but I've been reading the dart2js code and it looks like it should be a few lines of changes after implementing the stack trace mapper. I'm working on it.

konsultaner commented 1 month ago

@osa1 so for now the best would be to get test coverage by dart2js or is it realistic to hope for a fix on the chrome side any soon?

osa1 commented 1 month ago

@konsultaner I've reached out to the relevant team to get the Chromium issue triaged, but they seem to be away until the 18th. I will update when I hear back from them.

osa1 commented 1 month ago

@konsultaner see https://g-issues.chromium.org/issues/354020953#comment4. It looks like Chrome won't be supporting coverage for Wasm code any time soon.

I briefly looked at coverage handling of this package and there doesn't seem to be anything specific to JS, it should handle Wasm coverage output as well. I've updated the stack_frame package in the meantime but it seems like it wasn't necessary for coverage support in dart test.

In any case, we can get back to this once Chrome starts generating coverage info of Wasm code. I suspect it will just work without any changes.

konsultaner commented 1 month ago

@osa1 thanks for your investigation. Seems like we have to wait for the feature. Thx 😇

konsultaner commented 5 days ago

To anyone reading this issue: I guess showing to the chrome team, that there is demand for wasm code coverage support, is the best way to let it happen. Go to:

https://issues.chromium.org/issues/354020953

sign in with your google account and hit the +1 on the top right corner, if you are interested into this feature.

kevmoo commented 5 days ago

To anyone reading this issue: I guess showing to the chrome team, that there is demand for wasm code coverage support, is the best way to let it happen. Go to:

https://issues.chromium.org/issues/354020953

sign in with your google account and hit the +1 on the top right corner, if you are interested into this feature.

See the guy who filed that issue? I wonder who ke...@google.com is 😉