search

googleapis / sample-tester

Tool for testing semantically equivalent samples in multiple languages and environments
Apache License 2.0
10 stars 12 forks source link

Can't get `--cases FILTER` to work #115

Closed beccasaurus closed 5 years ago

beccasaurus commented 5 years ago

I'm trying to use --cases in 0.16.0 and I can't get the filtering to work.

My test suite includes lots of cases, including the following excerpt of suite and case names from the output:

  RUNNING: Test suite: "Transcript Audio File (Cloud Storage)"
    PASSED: Test case: "speech_transcribe_sync_gcs (no arguments)"
    PASSED: Test case: "speech_transcribe_sync_gcs (--storage_uri)"
  RUNNING: Test suite: "Separating different speakers (Local File) (LRO) (Beta)"
    PASSED: Test case: "speech_transcribe_diarization_beta (no arguments)"
    PASSED: Test case: "speech_transcribe_diarization_beta (--local_file_path)"

Example: I want to run some of the "diarization" tests

I tried the following:

$ sample-tester --cases diarization samples/

And this was the result:

$ sample-tester --cases diarization samples/
/Users/rebeccataylor/.pyenv/versions/3.7.3/lib/python3.7/site-packages/sampletester/parser.py:104: YAMLLoadWarning: calling yaml.load_all() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  self.add_documents(*[Document(file_name, doc) for doc in yaml.load_all(content)])
RUNNING: Test environment: ""
  RUNNING: Test suite: "Getting word timestamps (Cloud Storage) (LRO)"
    PASSED: Test case: "speech_transcribe_async_word_time_offsets_gcs (no arguments)"
^C
keyboard interrupt; aborting

It started to run all tests, notice that it ran cases without 'diarization' in the name.

I tried all of the following commands. Every one of them resulted in the same (all the tests ran, it did not filter which tests run as far as I could tell).

As far as I can tell, the --cases flag is completely ignored

vchudnov-g commented 5 years ago

@beccasaurus Since you're all set up, could you quickly check whether this is still happening with v0.16.1?

beccasaurus commented 5 years ago

Confirmed, still happening in 0.16.1