w3c / aria-at

Assistive Technology ARIA Experience Assessment
https://aria-at.netlify.app
Other
154 stars 28 forks source link

Build process non-deterministically produces invalid HTML files #910

Closed jugglinmike closed 1 year ago

jugglinmike commented 1 year ago

On my system (GNU/Linux, Ubuntu 18.04, Node.js 16.17.0), the npm build command occasionally outputs invalid HTML files.

$ git show --oneline --no-patch
b5fe3efd (HEAD -> master, upstream/master) Create tests for APG design pattern example: Horizontal Multi-Thumb Slider (#511)
$ git status
On branch master
Your branch is ahead of 'bocoup/master' by 174 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

Steps to reproduce:

  1. Build the HTML files five times, saving the output each time:

    $ for x in $(seq 5); do npm run build && mv build build-$x; done

  2. Search the five build directories for files with multiple closing HTML tags:

    $ grep -Ro '</html' build- | uniq -c | sort -n | grep -E '^\s[2-9]'

Expected behavior: the search returns zero results

Actual behavior: the search returns one or more results

For my trial, the search returned:

2 build-2/tests/command-button/reference/2022-4-8_124112/button.html:</html
2 build-3/tests/tabs-manual-activation/reference/2021-2-15_124757/tabs.html:</html
2 build-5/tests/form/reference/2021-10-24_152221/form.html:</html
2 build-5/tests/modal-dialog/reference/2022-4-7_15544/dialog.html:</html

The symptom is more severe than duplicated closing tags; that just happens to be a straightforward way to detect the problem.

Here's a comparison highlighting the invalidity in one of those files ```diff $ git diff -w --no-index build-1/tests/command-button/reference/2022-4-8_124112/button.html build-2/tests/command-button/reference/2022-4-8_124112/button.html diff --git a/build-1/tests/command-button/reference/2022-4-8_124112/button.html b/build-2/tests/command-button/reference/2022-4-8_124112/button.html index d17cbeb1..8d7a8451 100644 --- a/build-1/tests/command-button/reference/2022-4-8_124112/button.html +++ b/build-2/tests/command-button/reference/2022-4-8_124112/button.html @@ -7,23 +7,7 @@ - - - - +^M

Command Button Example

@@ -48,6 +32,18 @@ Navigate backwards from here
+ ^M + ^M + ^M +^M +^M +s from here +
+
Print Page
+
+ Navigate backwards from here +
+ ```

I've scraped all of the test HTML which is currently published on the production deployment of ARIA-AT App, and the good news is that it appears to be valid.

That may mean that the most recent build was lucky (notice how in my trial, the first and fourth build didn't include any duplicated closing HTML tags), or it may mean that this problem is specific to my machine. I'm wondering if anyone else can reproduce the behavior reported here.

mzgoddard commented 1 year ago

I can confirm this bug on my system.

shell output ```sh $ grep -Ro '
mzgoddard commented 1 year ago

Oh, I know what this is.

Two different file content are being written to the same file path. This is happening to each reference html file that has no setup script.

To fix this we need to not add the original reference html file to the build because we create one for the same file path with generated code.