Four tests fail in some environments

[carlsmedstad]

Hey :wave:

I'm the maintainer of the asciidoctor-pdf package in the Arch User Repositories. Recently, I introduced running the test suite as part of the package build process and multiple users are reporting the same failing tests:

==> Beginne check()...
...........................................................................................................................................................................F..F..................................................................................................................................................................................................................................................Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F.F................................

Failures:

  1) Asciidoctor::PDF::Converter - Audio should replace audio block with right pointer, path to audio file, and audio label
     Failure/Error: (expect pdf.lines).to eql expected_lines

       expected: ["before", "► /home/chris/.cache/pikaur/build/asciidoctor-pdf/src/asciidoctor-pdf-2.3.9/spec/fixtures/podcast.mp3 (audio)", "after"]
            got: ["before", "► /home/chris/.cache/pikaur/build/asciidoctor-pdf/src/asciidoctor-pdf-", "2.3.9/spec/fixtures/podcast.mp3 (audio)", "after"]

       (compared using eql?)
     # ./spec/audio_spec.rb:21:in `block (2 levels) in <top (required)>'

  2) Asciidoctor::PDF::Converter - Audio should show caption for audio if title is specified
     Failure/Error: (expect pdf.lines).to eql [%(\uf04b\u00a0#{fixture_file 'podcast-e1.mp3'} (audio)), 'Episode 1 of my podcast']

       expected: [" /home/chris/.cache/pikaur/build/asciidoctor-pdf/src/asciidoctor-pdf-2.3.9/spec/fixtures/podcast-e1.mp3 (audio)", "Episode 1 of my podcast"]
            got: [" /home/chris/.cache/pikaur/build/asciidoctor-pdf/src/asciidoctor-pdf-2.3.9/spec/fixtures/podcast-", "e1.mp3 (audio)", "Episode 1 of my podcast"]

       (compared using eql?)
     # ./spec/audio_spec.rb:56:in `block (2 levels) in <top (required)>'

  3) Asciidoctor::PDF::Converter - Video Local should replace video with video path and play icon if poster not specified
     Failure/Error: (expect pdf.lines).to eql [%(\uf04b\u00a0#{fixture_file 'asciidoctor.mp4'} (video))]

       expected: [" /home/chris/.cache/pikaur/build/asciidoctor-pdf/src/asciidoctor-pdf-2.3.9/spec/fixtures/asciidoctor.mp4 (video)"]
            got: [" /home/chris/.cache/pikaur/build/asciidoctor-pdf/src/asciidoctor-pdf-", "2.3.9/spec/fixtures/asciidoctor.mp4 (video)"]

       (compared using eql?)
     # ./spec/video_spec.rb:22:in `block (3 levels) in <top (required)>'

  4) Asciidoctor::PDF::Converter - Video Local should show caption for video with no poster if title is specified
     Failure/Error: (expect pdf.lines).to eql [%(\uf04b\u00a0#{fixture_file 'asciidoctor.mp4'} (video)), 'Asciidoctor training']

       expected: [" /home/chris/.cache/pikaur/build/asciidoctor-pdf/src/asciidoctor-pdf-2.3.9/spec/fixtures/asciidoctor.mp4 (video)", "Asciidoctor training"]
            got: [" /home/chris/.cache/pikaur/build/asciidoctor-pdf/src/asciidoctor-pdf-", "2.3.9/spec/fixtures/asciidoctor.mp4 (video)", "Asciidoctor training"]

       (compared using eql?)
     # ./spec/video_spec.rb:43:in `block (3 levels) in <top (required)>'

Finished in 4 minutes 19.1 seconds (files took 0.98803 seconds to load)
2141 examples, 4 failures

Failed examples:

rspec ./spec/audio_spec.rb:6 # Asciidoctor::PDF::Converter - Audio should replace audio block with right pointer, path to audio file, and audio label
rspec ./spec/audio_spec.rb:48 # Asciidoctor::PDF::Converter - Audio should show caption for audio if title is specified
rspec ./spec/video_spec.rb:15 # Asciidoctor::PDF::Converter - Video Local should replace video with video path and play icon if poster not specified
rspec ./spec/video_spec.rb:35 # Asciidoctor::PDF::Converter - Video Local should show caption for video with no poster if title is specified

==> FEHLER: Ein Fehler geschah in check().
    Breche ab...

I've not been able to reproduce this on my system. Are these tests known to be flaky? I suspect something with the users environment causes the strings in the tests to be split differently, but I'm out-of-depth as to figure out what.

For now I have removed the tests from the package, but if you have any ideas on how to fix them that would be greatly appreciated. Thanks!

[mojavelinux]

Are these tests known to be flaky?

No, they are not. They run nightly and on every commit in CI.

A possible problem is that the system encoding is not set to UTF-8. If the tests fail in CI, then I will address it. Otherwise, I'm going to assume the problem is a misconfiguration with the system running the tests.

[mojavelinux]

...upon further inspection, I may have another idea.

The path of the audio and video file is being inserted into the output. Since the path at which you are running the tests is so long, the path is getting wrapped. Hence, it's breaking the assertion of the test. I think the test may need to account for this situation.

[mojavelinux]

Tests always have to make some assumptions, and thus far this assumption has held up. But if you take the test suite well outside of its context, then these kinds of discrepancies inevitably will appear.