Design decisions / architecture documentation on tests & test runner

[louaybassbouss]

Design decision of Test Runners and Tests already made

• The main design principles of all WAVE Test Runners and Tests including DPCTF are drawn from WPT which includes three main elements:
  • The Test Server
  • The Test implementations
  • The Test Harness Client
• All these concepts are adopted in the WAVE Test Runners, with following additional extensions. All these extensions are discussed, agreed and merged back into WPT after a very long and intensive process (Pull Requests)
  • Support of embedded devices by moving application logic from client to server (thin client on embedded device, most application logic runs on server)
  • Support of companion page to configure, control and monitor test sessions
  • Support of REST API for using the Test Runner from external applications (e.g. OF)
• Split Test Runner and Test implementations in two difference repositories. This design decision is proposed by Fraunhofer and agreed in early DPCTF Testing Calls.
  • We apply here “Separation of Concerns” design principle. This way the DPCTF Test Runner can be easily updated if there is a new version of WPT.
• Containerization. This is also a design decision taken within DPCTF. Main design principle applied here is “modularization” driven by the fact, that WPT is very hard to setup and deploy.
  • We used docker as a container solution. Our main focus was on Linux, but Mac and windows are added as well. Question if we need to put efforts to support all platforms or only focus on Linux since this the platform for productive deployment and mark other platforms as “run on your own risk”.
• DPCTF specific design decisions for DPCTF tests:
  • Auto generation of tests (JS + HTML + Content) as combination of Test Implementation (JS + HTML) and Content
  • Use QR code as a mechanism to communicate test state to other systems (e.g. OF)

Lets use this issue to discuss these design decisions and if the there is a need to put them in a formal document (and keep it up-to-date in the future).

@jpiesing FYI

[javierbarellano4v]

@louaybassbouss First of all, thanks for posting these notes.

This posting is part of effort to confirm details on the inner workings of DPC test suite:

1. Generally speaking if I need more details on Test Server and Test Harness Client I should consult W3C's documentation on WPT?

2. Can we safely say that: a DPC test = Test Content + Instantiated HTML Template + JS ?

3. An instantiated HTML template contains logic for a specific use case, e.g., out of order loading, sequential track playback, for Type 3 playback?

4. Roughly speaking these interaction diagrams are still accurate?

      https://github.com/web-platform-tests/wpt/issues/16214

Just a couple to get started.... Thanks so much!

[jpiesing]

@louaybassbouss ?

This posting is part of effort to confirm details on the inner workings of DPC test suite:

1. Generally speaking if I need more details on Test Server and Test Harness Client I should consult W3C's documentation on WPT?

That is my understanding but @louaybassbouss needs to confirm.

2. Can we safely say that:   a DPC test = Test Content + Instantiated HTML Template + JS ?

Yes assuming what you mean by "Instantiated" is that the URLs for the Test Content are inserted in the original template.

3. An instantiated HTML template contains logic for a specific use case, e.g., out of order loading, sequential track playback, for Type 3 playback?

Yes-ish subject to previous comment re "instantiated'. This is all what's included in the original (un-instantiated?) template and copied over unmodified into what I think you are calling the "instantiated" template.

4. Roughly speaking these interaction diagrams are still accurate?
   ```
       https://github.com/web-platform-tests/wpt/issues/16214
   ```

@louaybassbouss ?

[javierbarellano4v]

Thanks @jpiesing

I need to redefine my 'instantiated template' as a HTML5 template populated or bound with URLs for Test Content.

And, 'un-instantiated' template or original template, is a HTML5 containing logic for specific use case, e.g., sequential track playback, for type 3 playback not bound to particular test content.

Think its more accurate, right?

[louaybassbouss]

Thanks @javierbarellano4v and @jpiesing sorry for delay I was on vacation I will reply in the next few hours to the comments, just landed in Vegas attending NAB.

[louaybassbouss]

This posting is part of effort to confirm details on the inner workings of DPC test suite:

1. Generally speaking if I need more details on Test Server and Test Harness Client I should consult W3C's documentation on WPT?

That is my understanding but @louaybassbouss needs to confirm.

I can confirm

2. Can we safely say that:   a DPC test = Test Content + Instantiated HTML Template + JS ?

Yes assuming what you mean by "Instantiated" is that the URLs for the Test Content are inserted in the original template.

agree with @jpiesing

3. An instantiated HTML template contains logic for a specific use case, e.g., out of order loading, sequential track playback, for Type 3 playback?

Yes-ish subject to previous comment re "instantiated'. This is all what's included in the original (un-instantiated?) template and copied over unmodified into what I think you are calling the "instantiated" template.

Yes the original un-instantiated template includes the test logic with placeholders for the URLs.

4. Roughly speaking these interaction diagrams are still accurate?

   https://github.com/web-platform-tests/wpt/issues/16214

Exactly the top one with "WPT Server (Python)" (attached below) and this is merged into WPT.

[javierbarellano4v]

@louaybassbouss Thanks, much appreciated! My apology for not replying sooner I was out of town on travel as well.

I may have additional questions later. I'm presenting DPCTF internally and thus all my questions.

Enjoy NAB in Vegas!

Thank you.

[gitwjr]

@javierbarellano4v - please confirm this is resolved. @FritzHeiden documented in the TR. If no further comments, we can close.

[javierbarellano4v]

This is resolved. Please close issue