search

Tracktion / pluginval

Cross platform plugin testing and validation tool
GNU General Public License v3.0
390 stars 57 forks source link

pluginval will run forever when timeout_ms is set to -1 #114

Closed zsliu98 closed 5 months ago

zsliu98 commented 1 year ago

See this action where I set timeout as an environment variable:

    - name: Pluginval VST3 validations
      working-directory: ${{ env.BUILD_DIR }}
      shell: bash
      run: |
        ${{ matrix.pluginval-binary }} --verbose --validate "${{ env.VST3_PATH }}"
      env:
        STRICTNESS_LEVEL: 10
        TIMEOUT_MS: -1
        REPEAT: 4

and this action where I specify it in the command:

    - name: Pluginval VST3 validations
      working-directory: ${{ env.BUILD_DIR }}
      shell: bash
      run: |
        ${{ matrix.pluginval-binary }} --verbose --timeout-ms -1 --validate "${{ env.VST3_PATH }}"
      env:
        STRICTNESS_LEVEL: 10
        REPEAT: 4

In both cases, pluginval will output some welcome messages and run forever (> 5 mins without output). I am not sure whether it is the correct way to specify the timeout. Any ideas?

drowaudio commented 1 year ago

A timeout of -1 means "run forever": From the --help command:

By default this is 30s but can be set to -1 to never timeout

It sounds like your plugin is hanging if no output is being produced? Have you tried running it in the debugger under pluginval? https://github.com/Tracktion/pluginval/blob/develop/docs/Debugging%20a%20failed%20validation.md

zsliu98 commented 1 year ago 
Run ./pluginval --verbose --timeout-ms -1 --validate "ZLTest_artefacts/Release/VST3/ZL Test.vst3"
  ./pluginval --verbose --timeout-ms -1 --validate "ZLTest_artefacts/Release/VST3/ZL Test.vst3"
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    PROJECT_NAME: ZLTest
    PRODUCT_NAME: ZL Test
    BUILD_TYPE: Release
    BUILD_DIR: Builds
    GITHUB_TOKEN: ***
    DISPLAY: :0
    CMAKE_BUILD_PARALLEL_LEVEL: 3
    HOMEBREW_NO_INSTALL_CLEANUP: 1
    ARTIFACTS_PATH: Builds/ZLTest_artefacts/Release
    VST3_PATH: ZLTest_artefacts/Release/VST3/ZL Test.vst3
    AU_PATH: ZLTest_artefacts/Release/AU/ZL Test.component
    AUV3_PATH: ZLTest_artefacts/Release/AUv3/ZL Test.appex
    INSTALLER_NAME: ZLTest-0.0.0
    STRICTNESS_LEVEL: 10
    REPEAT: 4
//==============================================================================
pluginval
JUCE v7.0.[2](https://github.com/ZL-Audio/ZLTest/actions/runs/6038217096/job/16384115802#step:18:2)

Description: 
  Validate plugins to test compatibility with hosts and verify plugin API conformance

Usage: 
  --validate [pathToPlugin]
    Validates the plugin at the given path.
    N.B. the --validate flag is optional if the path is the last argument. This enables you to validate a plugin with simply "pluginval path_to_plugin".
  --strictness-level [1-10]
    Sets the strictness level to use. A minimum level of 5 (also the default) is recomended for compatibility. Higher levels include longer, more thorough tests such as fuzzing.
  --random-seed [hex or int]
    Sets the random seed to use for the tests. Useful for replicating test environments.
  --timeout-ms [numMilliseconds]
    Sets a timout which will stop validation with an error if no output from any test has happened for this number of ms.
    By default this is [3](https://github.com/ZL-Audio/ZLTest/actions/runs/6038217096/job/16384115802#step:18:3)0s but can be set to -1 to never timeout.
  --verbose
    If specified, outputs additional logging information. It can be useful to turn this off when building with CI to avoid huge log files.
  --skip-gui-tests
    If specified, avoids tests that create GUI windows, which can cause problems on headless CI systems.
  --repeat [num repeats]
    If specified repeats the tests a given number of times. Note that this does not delete and re-instantiate the plugin for each repeat.  --randomise
    If specified the tests are run in a random order per repeat.  --data-file [pathToFile]
    If specified, sets a path to a data file which can be used by tests to configure themselves. This can be useful for things like known audio output.
  --output-dir [pathToDir]
    If specified, sets a directory to store the log files. This can be useful for continuous integration.
  --disabled-tests [pathToFile]
    If specified, sets a path to a file that should have the names of disabled tests on each row.
  --sample-rates [list of comma separated sample rates]
    If specified, sets the list of sample rates at which tests will be executed (default=[4](https://github.com/ZL-Audio/ZLTest/actions/runs/6038217096/job/16384115802#step:18:4)4100,48000,96000)
  --block-sizes [list of comma separated block sizes]
    If specified, sets the list of block sizes at which tests will be executed (default=64,128,2[5](https://github.com/ZL-Audio/ZLTest/actions/runs/6038217096/job/16384115802#step:18:5)[6](https://github.com/ZL-Audio/ZLTest/actions/runs/6038217096/job/16384115802#step:18:6),512,[10](https://github.com/ZL-Audio/ZLTest/actions/runs/6038217096/job/16384115802#step:18:10)[24](https://github.com/ZL-Audio/ZLTest/actions/runs/6038217096/job/16384115802#step:18:25))
  --vst3validator [pathToValidator]
    If specified, this will run the VST3 validator as part of the test process.
  --version
    Print pluginval version.

Exit code: 
  0 if all tests complete successfully
  1 if there are any errors

Additionally, you can specify any of the command line options as environment varibles by removing prefix dashes, converting internal dashes to underscores and captialising all letters e.g. "--skip-gui-tests" > "SKIP_GUI_TESTS=1", "--timeout-ms [30](https://github.com/ZL-Audio/ZLTest/actions/runs/6038217096/job/16384115802#step:18:31)000" > "TIMEOUT_MS=30000"
Specifying specific command-line options will override any environment variables set for that option.

 pluginval --version                  Prints the current version number
 pluginval --help|-h                  Prints the list of commands
 pluginval --validate [pathToPlugin]  Validates the file (or IDs for AUs).
 pluginval --run-tests                Runs the internal unit tests.

Error: The operation was canceled.
drowaudio commented 5 months ago

I think the problem is just that the -1 argument isn't parsed correctly and is seen as two tokens, - and 1 so it only take the - and fails. I think if you quote the "-1" it will work as expected (I've updated the docs to reflect this).