Support Microsoft Speech SDK

[liankuai]

Please help us evaluate our speech recognition, it support Microsoft Speech SDK include en, zh, thank you very much.

[dophist]

Thanks for the submission. One comment from a quick review:

please refer to https://github.com/SpeechColab/Leaderboard/blob/master/HOW_TO_SUBMIT.md , to organize the calling script into a self-contained model-image(you can refer to existing microsoft model dir and copy necessary resources) , and validate the model image can actually work with the pipeline. So I can directly start benchmark instead of reorg your stuff and debug your code.

[dophist]

And maybe, give the submitted model-image an ID like something "microsoft_sdkapi{zh,en}", your call.

[liankuai]

I got it , I will improve it.

[dophist]

No worries. By the way, you can just skip those "upload to cloud model-zoo" stuff in the guideline, and complete the model submission entirely via this pull request, and don't hesitate to ask if anything in the guideline is unclear.

[liankuai]

Hi, I have prepared model-image via this pull request , then can I need to validate prepared model-image with leaderboard pipeline on my local machine. @dophist

[dophist]

Looks great, thanks. You mean you have already passed the validation ? (validation steps are described in https://github.com/SpeechColab/Leaderboard/blob/master/HOW_TO_SUBMIT.md section 1.7 )

[liankuai]

I am verifying, I will call you after passing. @dophist

[liankuai]

When I execute DockerFile, the system prompts "No such file or direcitory". this is the command:

docker run --user "$(id -u):$(id -g)" -e LEADERBOARD=/app/ speechio/leaderboard -v /Leaderboard:/app/speechio/leaderboard speechio/leaderboard:microsoft_sdk_api_en /app/speechio/leaderboard/utils/benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI" . @dophist

[dophist]

right after the -v option，the path to Leaderboard is somehow incomplete，have a check why there is no prefix to the path

发自我的iPhone

------------------ Original ------------------ From: Hui @.> Date: Tue,Oct 26,2021 6:00 PM To: SpeechColab/Leaderboard @.> Cc: Jiayu DU @.>, Mention @.> Subject: Re: [SpeechColab/Leaderboard] Support Microsoft Speech SDK (PR #11)

When I execute DockerFile, the system prompts "No such file or direcitory". this is the command:

docker run --user "$(id -u):$(id -g)" -e LEADERBOARD=/app/ speechio/leaderboard -v /Leaderboard:/app/speechio/leaderboard speechio/leaderboard:microsoft_sdk_api_en /app/speechio/leaderboard/utils/benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI" .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[liankuai]

If I removed " /app/speechio/leaderboard/utils/benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI" ", I successfully build the docker container, if I call benchmark.sh directly, it also Failed, my command " sh benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI" ", it prompt "Syntax error: word unexpected (expecting "do") " @dophist

[dophist]

If I removed " /app/speechio/leaderboard/utils/benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI" ", I successfully build the docker container, if I call benchmark.sh directly, it also Failed, my command " sh benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI" ", it prompt "Syntax error: word unexpected (expecting "do") " @dophist

I suspect the default shell of your linux is not bash anymore. You shouldn't call this directly anyway, I would suggest to check why -v option failed to get the prefix path of "/Leaderboard"

[dophist]

If I removed " /app/speechio/leaderboard/utils/benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI" ", I successfully build the docker container, if I call benchmark.sh directly, it also Failed, my command " sh benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI" ", it prompt "Syntax error: word unexpected (expecting "do") " @dophist

I suspect the default shell of your linux is not bash anymore. You shouldn't call this directly anyway, I would suggest to check why -v option failed to get the prefix path of "/Leaderboard"

[liankuai]

Hi, the code: Leaderboard= os.getcwd(), I run it like this: “ python3 ops/leaderboard_runner requests/microsoft_sdk_api_en_request.yaml” in leaderboard repo dir，and leaderboard dir is root dir, So -v option failed to get the prefix path of "/Leaderboard". Is there something wrong here ?

[dophist]

you can set the path manually for the moment and check if you can pass the MINI validation, and leave the other thing to me. By the way, what's your local operating system?

[liankuai]

My local operating system is ubuntu 20.04. I manually checked the container I created, which contains all the content of the host. But benchmark.sh needs to be removed before the container can be created successfully. I run it like this : docker run -d -p 9000:9000 -it --name speech_microsoft_sdk_api_en_env_vol --env LEADERBOARD=/app/speechio/leaderboard -v /Leaderboard:/app/speechio/leaderboard speechio/leaderboard:microsoft_sdk_api_en

[dophist]

You said os.getcwd() can't not parse the prefix of '/Leaderboard' and now with "-v /Leaderboard" you can successfully mount the leaderboard dir ?

[liankuai]

I debug os.getcwd() ==/Leaderboard , if I removed the benchmark.sh on docker command, I can successfully mount the leaderboard dir. if add benchmark.sh, it will fail ,so suspect the benchmark.sh.

[liankuai]

Removed docker run command /app/speechio/leaderboard/utils/benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI"

[dophist]

paste your full command of container instantiation here, you should be able to get the command from logged output of leaderboard_runner

[liankuai]

2021-10-26 19:21:06,057 [INFO] Pulling up benchmarking container: docker run --user "$(id -u):$(id -g)" -e LEADERBOARD=/app/speechio/leaderboard -v /Leaderboard:/app/speechio/leaderboard speechio/leaderboard:microsoft_sdk_api_en /app/speechio/leaderboard/utils/benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI"

[dophist]

2021-10-26 19:21:06,057 [INFO] Pulling up benchmarking container: docker run --user "$(id -u):$(id -g)" -e LEADERBOARD=/app/speechio/leaderboard -v /Leaderboard:/app/speechio/leaderboard speechio/leaderboard:microsoft_sdk_api_en /app/speechio/leaderboard/utils/benchmark.sh --stage 0 --max-num-utts 10000 microsoft_sdk_api_en "MINI"

and what's your actual prefix path of "/Leaderboard"? you can have a try to replace their with correct path, and run the fixed full command in terminal, and see what happens

[liankuai]

Hi, My email is "v-huilia@microsoft.com" and my wechat is "liankuai", Can I call with you by them?

[dophist]

Hi, My email is "v-huilia@microsoft.com" and my wechat is "liankuai", Can I call with you by them?

sure, I added you via wechat