Open klahnakoski opened 6 years ago
ActiveData has a download limit of 50K records. So getting the full file list may be more difficult than it is worth. Instead, find the original artifact and download it.
Here is the firefox-files
table: http://activedata.allizom.org/tools/query.html#query_id=lGdtmxm+
The etl
property has details about how the record traveled through the ETL pipeline, including sources used to populate it: http://activedata.allizom.org/tools/query.html#query_id=he3n3mcq
We can see recent artifacts with a query to the task
table: http://activedata.allizom.org/tools/query.html#query_id=F+y2Bj11
There is the LCOV rewriter that was written for this purpose and applies the most precise mapping we can have.
The LCOV rewriter requires a local copy of hg yes? If this is to be done by ActiveData's ETL pipeline, it would be preferable to have less dependencies.
I believe we discussed this rewriting is possible as yet-another-post-collection-step on the TC machines. Then the artifacts would have the correct filenames before they enter the pipeline. If it is done on TC, it should not loose the test-level detail we plan to track: https://treeherder.mozilla.org/#/jobs?repo=try&revision=182b519971d39b8c66c6faf61f62b87973d81608
I believe it is best that the translation is done in the ETL pipeline: It keeps the data translation in one place, the translation has less code dependencies, and the coverage generation stays as simple as possible.
Sorry, my last link was not right: Here is the job that generates per-test artifacts: https://treeherder.mozilla.org/#/jobs?repo=try&revision=182b519971d39b8c66c6faf61f62b87973d81608&selectedJob=145407889
The LCOV rewriter requires a local copy of hg yes? If this is to be done by ActiveData's ETL pipeline, it would be preferable to have less dependencies.
I believe we discussed this rewriting is possible as yet-another-post-collection-step on the TC machines. Then the artifacts would have the correct filenames before they enter the pipeline. If it is done on TC, it should not loose the test-level detail we plan to track: https://treeherder.mozilla.org/#/jobs?repo=try&revision=182b519971d39b8c66c6faf61f62b87973d81608
Yes, the mapping should work no matter if we run all tests or if we run only a subset of them.
I believe it is best that the translation is done in the ETL pipeline: It keeps the data translation in one place, the translation has less code dependencies, and the coverage generation stays as simple as possible.
We'd still translate in the ETL pipeline, but we'd translate using the mapping generated by the coverage task (uploaded as an artifact).
The jsdcov filenames are wrong. They are dynamically loaded and given a variety of path prefixes
Use the
firefox-files
table in ActiveData to map these filenames to the matching version found in the repo. I imagine the best-matching-suffix-search will do nicely. It is important this mapping is very fast.Please update
jsvm_to_es
to use this mapping