Design and implement robust testing story for single-file

[vitek-karas]

Historically single-file feature has been a source of expensive bugs. Unfortunately not just small bugs but large experience breaking bugs, which are discovered relatively late and thus more expensive to fix. The core of the problem is that single-file is a feature which depends on lot of very specific behaviors across large number of components and thus it's more prone to breaks from seemingly unrelated changes.

To solve this we need a robust testing story for single-file. It should include:

• Unit-test-like coverage in the runtime repo close to the specific components (host, runtime, sdk-task)
• Validation for SDK components to single-file (RID, runtime pack restore, picking the right host, errors for unsupported scenarios)
• Integration tests which validate the feature end-to-end
  • With sufficiently large platform/architecture matrix (for example frequent source of bugs are macos or arm64 targets)
  • Coverage for most options (self-contained true/false, with native library embedded/not, compressed on/off)
  • Reasonable coverage of cross feature integration tests
  • Trimmed + single-file
  • R2R + single-file
  • Behavior of common SDK commands with single-file project (build, run, test, watch)
• Subset of existing library tests to run in single-file mode

The exact location and type of these tests is TBD (expectation is spread between runtime and sdk repos)

[ghost]

Tagging subscribers to this area: @agocke, @vitek-karas, @vsadov See info in area-owners.md if you want to be subscribed.

Issue Details

---

Historically single-file feature has been a source of expensive bugs. Unfortunately not just small bugs but large experience breaking bugs, which are discovered relatively late and thus more expensive to fix. The core of the problem is that single-file is a feature which depends on lot of very specific behaviors across large number of components and thus it's more prone to breaks from seemingly unrelated changes. To solve this we need a robust testing story for single-file. It should include: * Unit-test-like coverage in the runtime repo close to the specific components (host, runtime, sdk-task) * Validation for SDK components to single-file (RID, runtime pack restore, picking the right host, errors for unsupported scenarios) * Integration tests which validate the feature end-to-end * With sufficiently large platform/architecture matrix (for example frequent source of bugs are macos or arm64 targets) * Coverage for most options (self-contained true/false, with native library embedded/not, compressed on/off) * Reasonable coverage of cross feature integration tests * Trimmed + single-file * R2R + single-file * Behavior of common SDK commands with single-file project (build, run, test, watch) * Subset of existing library tests to run in single-file mode The exact location and type of these tests is TBD (expectation is spread between runtime and sdk repos)

Author:	vitek-karas
Assignees:	elinor-fung
Labels:	`test-enhancement`, `area-Single-File`
Milestone:	-