search

biemster / gasr

Google Chrome SODA Offline Speech Recognition command line client
https://hackaday.io/project/164399-android-offline-speech-recognition-natively-on-pc
150 stars 18 forks source link

cannot reproduce the result #5

Closed benhuang2018 closed 3 years ago

benhuang2018 commented 3 years ago

Hi Sir/Madam,

I'm new to hackaday, i found this project very interesting, and i'm trying to reproduce the result with the instruction in description and readme file. but i can't make it right... please enlight me if i did something wrong...

my environment is ubuntu 18.04.5, it's a clean environment, i install it on a new drive.

here is my steps:

  1. download chrome beta from google (https://www.google.com/chrome/beta/), the version i got is 90.0.4430.51(64-bit)

  2. enable live caption from setting:

    setting -> advanced -> accessibility -> Caption -> live caption -> set to ON and wait for download complate

  3. clone the gasr repo: git clone https://github.com/biemster/gasr.git

  4. copy the library file from chrome's profile folder:

    cd /home//.config/google-chrome-beta/SODA/1.0.3/SODAFiles

    cp libsoda.so

  5. copy language file from chrome's profile folder:

    cd /home//.config/google-chrome-beta/SODALanguagePacks/en-US/1.0.0

    cp -r SODAModels/

  6. build the gasr.c

    make

    result >>> g++ -o gasr gasr.c -L. -Wl,-rpath,. -lsoda

  7. run the model with sample audio (https://raw.githubusercontent.com/Azure-Samples/cognitive-services-speech-sdk/f9807b1079f3a85f07cbb6d762c6b5449d536027/samples/cpp/windows/console/samples/whatstheweatherlike.wav):

    sudo ./gasr < samples_cpp_windows_console_samples_whatstheweatherlike.wav

    =====================================================================

    and my output is below:

sudo ./gasr < samples_cpp_windows_console_samples_whatstheweatherlike.wav

WARNING: Logging before InitGoogle() is written to STDERR W0406 11:03:00.264677 22156 soda_async_impl.cc:282] Creating soda_impl I0406 11:03:00.264819 22156 soda_impl.cc:285] Maximum audio history (ms): 30000 I0406 11:03:00.264840 22156 soda_impl.cc:306] Adding Resampler from 16000 to 16000 I0406 11:03:00.264977 22156 soda_impl.cc:506] Enabling power evaluator. I0406 11:03:00.264983 22156 soda_impl.cc:516] Adding preamble processor. I0406 11:03:00.264986 22156 soda_impl.cc:525] Enabling On Device ASR I0406 11:03:00.265156 22156 terse_processor.cc:707] Config file: ./SODAModels/configs/ONDEVICE_MEDIUM_CONTINUOUS.config I0406 11:03:00.265403 22156 terse_processor.cc:172] Loaded PipelineDef. I0406 11:03:00.265416 22156 dir_path.cc:52] Checking FileExists: ./endtoendmodel/marble_rnnt_model.syms.compact I0406 11:03:00.265421 22156 dir_path.cc:57] Not Found FileExists: ./endtoendmodel/marble_rnnt_model.syms.compact I0406 11:03:00.265424 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model.syms.compact I0406 11:03:00.265429 22156 dir_path.cc:54] Found FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model.syms.compact I0406 11:03:00.265547 22156 dir_path.cc:52] Checking FileExists: ./endtoendmodel/marble_rnnt_dictation_frontend_params.mean_stddev I0406 11:03:00.265555 22156 dir_path.cc:57] Not Found FileExists: ./endtoendmodel/marble_rnnt_dictation_frontend_params.mean_stddev I0406 11:03:00.265557 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/endtoendmodel/marble_rnnt_dictation_frontend_params.mean_stddev I0406 11:03:00.265561 22156 dir_path.cc:54] Found FileExists: ./SODAModels/endtoendmodel/marble_rnnt_dictation_frontend_params.mean_stddev I0406 11:03:00.265602 22156 dir_path.cc:52] Checking FileExists: ./endtoendmodel/marble_rnnt_model.wpm.portable I0406 11:03:00.265608 22156 dir_path.cc:57] Not Found FileExists: ./endtoendmodel/marble_rnnt_model.wpm.portable I0406 11:03:00.265610 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model.wpm.portable I0406 11:03:00.265614 22156 dir_path.cc:54] Found FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model.wpm.portable I0406 11:03:00.268811 22156 dir_path.cc:52] Checking FileExists: ./endtoendmodel/marble_rnnt_model.word_classifier I0406 11:03:00.269136 22156 dir_path.cc:57] Not Found FileExists: ./endtoendmodel/marble_rnnt_model.word_classifier I0406 11:03:00.269139 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model.word_classifier I0406 11:03:00.269145 22156 dir_path.cc:54] Found FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model.word_classifier I0406 11:03:00.269186 22156 dir_path.cc:52] Checking FileExists: ./denorm/embedded_replace_annotated_punct_words_dash.mfar I0406 11:03:00.269193 22156 dir_path.cc:57] Not Found FileExists: ./denorm/embedded_replace_annotated_punct_words_dash.mfar I0406 11:03:00.269195 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/denorm/embedded_replace_annotated_punct_words_dash.mfar I0406 11:03:00.269200 22156 dir_path.cc:54] Found FileExists: ./SODAModels/denorm/embedded_replace_annotated_punct_words_dash.mfar I0406 11:03:00.269240 22156 dir_path.cc:52] Checking FileExists: ./denorm/embedded_fix_ampm.mfar I0406 11:03:00.269246 22156 dir_path.cc:57] Not Found FileExists: ./denorm/embedded_fix_ampm.mfar I0406 11:03:00.269248 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/denorm/embedded_fix_ampm.mfar I0406 11:03:00.269252 22156 dir_path.cc:54] Found FileExists: ./SODAModels/denorm/embedded_fix_ampm.mfar I0406 11:03:00.269277 22156 dir_path.cc:52] Checking FileExists: ./denorm/embedded_class_denorm.mfar I0406 11:03:00.269284 22156 dir_path.cc:57] Not Found FileExists: ./denorm/embedded_class_denorm.mfar I0406 11:03:00.269287 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/denorm/embedded_class_denorm.mfar I0406 11:03:00.269291 22156 dir_path.cc:54] Found FileExists: ./SODAModels/denorm/embedded_class_denorm.mfar I0406 11:03:00.269358 22156 dir_path.cc:52] Checking FileExists: ./denorm/embedded_normalizer.mfar I0406 11:03:00.269363 22156 dir_path.cc:57] Not Found FileExists: ./denorm/embedded_normalizer.mfar I0406 11:03:00.269366 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/denorm/embedded_normalizer.mfar I0406 11:03:00.269369 22156 dir_path.cc:54] Found FileExists: ./SODAModels/denorm/embedded_normalizer.mfar I0406 11:03:00.269407 22156 dir_path.cc:52] Checking FileExists: ./denorm/porn_normalizer_on_device.mfar I0406 11:03:00.269413 22156 dir_path.cc:57] Not Found FileExists: ./denorm/porn_normalizer_on_device.mfar I0406 11:03:00.269415 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/denorm/porn_normalizer_on_device.mfar I0406 11:03:00.269419 22156 dir_path.cc:54] Found FileExists: ./SODAModels/denorm/porn_normalizer_on_device.mfar I0406 11:03:00.269465 22156 dir_path.cc:52] Checking FileExists: ./acousticmodel/MARBLE_DICTATION_EP.endpointer_portable_lstm_model I0406 11:03:00.269472 22156 dir_path.cc:57] Not Found FileExists: ./acousticmodel/MARBLE_DICTATION_EP.endpointer_portable_lstm_model I0406 11:03:00.269475 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/acousticmodel/MARBLE_DICTATION_EP.endpointer_portable_lstm_model I0406 11:03:00.269480 22156 dir_path.cc:54] Found FileExists: ./SODAModels/acousticmodel/MARBLE_DICTATION_EP.endpointer_portable_lstm_model I0406 11:03:00.269483 22156 neural_network_resource.cc:71] Initializing for TENSORFLOW_LITE I0406 11:03:00.269643 22156 dir_path.cc:52] Checking FileExists: ./acousticmodel/MARBLE_DICTATION_EP.endpointer_portable_lstm_mean_stddev I0406 11:03:00.269650 22156 dir_path.cc:57] Not Found FileExists: ./acousticmodel/MARBLE_DICTATION_EP.endpointer_portable_lstm_mean_stddev I0406 11:03:00.269653 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/acousticmodel/MARBLE_DICTATION_EP.endpointer_portable_lstm_mean_stddev I0406 11:03:00.269658 22156 dir_path.cc:54] Found FileExists: ./SODAModels/acousticmodel/MARBLE_DICTATION_EP.endpointer_portable_lstm_mean_stddev I0406 11:03:00.269678 22156 dir_path.cc:52] Checking FileExists: ./magic_mic/MARBLE_V2_acoustic_model.int8.tflite I0406 11:03:00.269683 22156 dir_path.cc:57] Not Found FileExists: ./magic_mic/MARBLE_V2_acoustic_model.int8.tflite I0406 11:03:00.269686 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/magic_mic/MARBLE_V2_acoustic_model.int8.tflite I0406 11:03:00.269691 22156 dir_path.cc:54] Found FileExists: ./SODAModels/magic_mic/MARBLE_V2_acoustic_model.int8.tflite I0406 11:03:00.269694 22156 neural_network_resource.cc:71] Initializing for TENSORFLOW_LITE I0406 11:03:00.269802 22156 dir_path.cc:52] Checking FileExists: ./magic_mic/MARBLE_V2_acoustic_meanstddev_vector I0406 11:03:00.269808 22156 dir_path.cc:57] Not Found FileExists: ./magic_mic/MARBLE_V2_acoustic_meanstddev_vector I0406 11:03:00.269811 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/magic_mic/MARBLE_V2_acoustic_meanstddev_vector I0406 11:03:00.269816 22156 dir_path.cc:54] Found FileExists: ./SODAModels/magic_mic/MARBLE_V2_acoustic_meanstddev_vector I0406 11:03:00.269832 22156 dir_path.cc:52] Checking FileExists: ./magic_mic/MARBLE_V2_vocabulary.syms I0406 11:03:00.269836 22156 dir_path.cc:57] Not Found FileExists: ./magic_mic/MARBLE_V2_vocabulary.syms I0406 11:03:00.269839 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/magic_mic/MARBLE_V2_vocabulary.syms I0406 11:03:00.269844 22156 dir_path.cc:54] Found FileExists: ./SODAModels/magic_mic/MARBLE_V2_vocabulary.syms I0406 11:03:00.271909 22156 dir_path.cc:52] Checking FileExists: ./magic_mic/MARBLE_V2_model.int8.tflite I0406 11:03:00.271918 22156 dir_path.cc:57] Not Found FileExists: ./magic_mic/MARBLE_V2_model.int8.tflite I0406 11:03:00.271921 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/magic_mic/MARBLE_V2_model.int8.tflite I0406 11:03:00.271925 22156 dir_path.cc:54] Found FileExists: ./SODAModels/magic_mic/MARBLE_V2_model.int8.tflite I0406 11:03:00.273563 22156 dir_path.cc:52] Checking FileExists: ./endtoendmodel/marble_rnnt_model-encoder.part_0.tflite I0406 11:03:00.273572 22156 dir_path.cc:57] Not Found FileExists: ./endtoendmodel/marble_rnnt_model-encoder.part_0.tflite I0406 11:03:00.273575 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model-encoder.part_0.tflite I0406 11:03:00.273580 22156 dir_path.cc:54] Found FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model-encoder.part_0.tflite I0406 11:03:00.273582 22156 neural_network_resource.cc:71] Initializing for PLAIN_TENSORFLOW_LITE I0406 11:03:00.279180 22156 dir_path.cc:52] Checking FileExists: ./endtoendmodel/marble_rnnt_model-encoder.part_1.tflite I0406 11:03:00.279198 22156 dir_path.cc:57] Not Found FileExists: ./endtoendmodel/marble_rnnt_model-encoder.part_1.tflite I0406 11:03:00.279201 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model-encoder.part_1.tflite I0406 11:03:00.279207 22156 dir_path.cc:54] Found FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model-encoder.part_1.tflite I0406 11:03:00.279209 22156 neural_network_resource.cc:71] Initializing for PLAIN_TENSORFLOW_LITE I0406 11:03:00.297939 22156 dir_path.cc:52] Checking FileExists: ./endtoendmodel/marble_rnnt_model-rnnt.decoder.tflite I0406 11:03:00.297966 22156 dir_path.cc:57] Not Found FileExists: ./endtoendmodel/marble_rnnt_model-rnnt.decoder.tflite I0406 11:03:00.297971 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model-rnnt.decoder.tflite I0406 11:03:00.297980 22156 dir_path.cc:54] Found FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model-rnnt.decoder.tflite I0406 11:03:00.297984 22156 neural_network_resource.cc:71] Initializing for PLAIN_TENSORFLOW_LITE I0406 11:03:00.304153 22156 dir_path.cc:52] Checking FileExists: ./endtoendmodel/marble_rnnt_model-rnnt.joint.tflite I0406 11:03:00.304173 22156 dir_path.cc:57] Not Found FileExists: ./endtoendmodel/marble_rnnt_model-rnnt.joint.tflite I0406 11:03:00.304176 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model-rnnt.joint.tflite I0406 11:03:00.304181 22156 dir_path.cc:54] Found FileExists: ./SODAModels/endtoendmodel/marble_rnnt_model-rnnt.joint.tflite I0406 11:03:00.304184 22156 neural_network_resource.cc:71] Initializing for PLAIN_TENSORFLOW_LITE I0406 11:03:00.305925 22156 dir_path.cc:52] Checking FileExists: ./voice_match/MARBLE_speakerid.tflite I0406 11:03:00.305935 22156 dir_path.cc:57] Not Found FileExists: ./voice_match/MARBLE_speakerid.tflite I0406 11:03:00.305937 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/voice_match/MARBLE_speakerid.tflite I0406 11:03:00.305941 22156 dir_path.cc:54] Found FileExists: ./SODAModels/voice_match/MARBLE_speakerid.tflite I0406 11:03:00.309580 22156 terse_processor.cc:189] Initialized ResourceManager. I0406 11:03:00.309700 22156 terse_processor.cc:200] Initialized GoogleRecognizer. I0406 11:03:00.309708 22156 context-module-factory.cc:35] ContextModuleFactory: Initializing ContextModule. I0406 11:03:00.309824 22156 dir_path.cc:52] Checking FileExists: ./context_prebuilt/apps.txt I0406 11:03:00.309837 22156 dir_path.cc:57] Not Found FileExists: ./context_prebuilt/apps.txt I0406 11:03:00.309842 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/context_prebuilt/apps.txt I0406 11:03:00.309847 22156 dir_path.cc:54] Found FileExists: ./SODAModels/context_prebuilt/apps.txt I0406 11:03:00.309888 22156 dir_path.cc:52] Checking FileExists: ./context_prebuilt/en-US_android-auto_car_automation.action.union_STD_FST.fst I0406 11:03:00.309894 22156 dir_path.cc:57] Not Found FileExists: ./context_prebuilt/en-US_android-auto_car_automation.action.union_STD_FST.fst I0406 11:03:00.309897 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/context_prebuilt/en-US_android-auto_car_automation.action.union_STD_FST.fst I0406 11:03:00.309902 22156 dir_path.cc:54] Found FileExists: ./SODAModels/context_prebuilt/en-US_android-auto_car_automation.action.union_STD_FST.fst I0406 11:03:00.328810 22156 dir_path.cc:52] Checking FileExists: ./context_prebuilt/contacts.txt I0406 11:03:00.328837 22156 dir_path.cc:57] Not Found FileExists: ./context_prebuilt/contacts.txt I0406 11:03:00.328842 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/context_prebuilt/contacts.txt I0406 11:03:00.328866 22156 dir_path.cc:54] Found FileExists: ./SODAModels/context_prebuilt/contacts.txt I0406 11:03:00.329062 22156 dir_path.cc:52] Checking FileExists: ./context_prebuilt/songs.txt I0406 11:03:00.329071 22156 dir_path.cc:57] Not Found FileExists: ./context_prebuilt/songs.txt I0406 11:03:00.329073 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/context_prebuilt/songs.txt I0406 11:03:00.329077 22156 dir_path.cc:54] Found FileExists: ./SODAModels/context_prebuilt/songs.txt I0406 11:03:00.329116 22156 dir_path.cc:52] Checking FileExists: ./context_prebuilt/en-US_android-auto_top_radio_station_frequencies_STD_FST.fst I0406 11:03:00.329122 22156 dir_path.cc:57] Not Found FileExists: ./context_prebuilt/en-US_android-auto_top_radio_station_frequencies_STD_FST.fst I0406 11:03:00.329125 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/context_prebuilt/en-US_android-auto_top_radio_station_frequencies_STD_FST.fst I0406 11:03:00.329143 22156 dir_path.cc:54] Found FileExists: ./SODAModels/context_prebuilt/en-US_android-auto_top_radio_station_frequencies_STD_FST.fst I0406 11:03:00.329230 22156 dir_path.cc:52] Checking FileExists: ./context_prebuilt/en-US_android-auto_manual_fixes_STD_FST.fst I0406 11:03:00.329238 22156 dir_path.cc:57] Not Found FileExists: ./context_prebuilt/en-US_android-auto_manual_fixes_STD_FST.fst I0406 11:03:00.329240 22156 dir_path.cc:52] Checking FileExists: ./SODAModels/context_prebuilt/en-US_android-auto_manual_fixes_STD_FST.fst I0406 11:03:00.329244 22156 dir_path.cc:54] Found FileExists: ./SODAModels/context_prebuilt/en-US_android-auto_manual_fixes_STD_FST.fst W0406 11:03:00.329525 22156 terse_processor.cc:1753] SODA could not build Hotquery Matcher. W0406 11:03:00.329545 22156 terse_processor.cc:288] TISID disabled. I0406 11:03:00.329572 22156 terse_processor.cc:809] Domain: CAPTION I0406 11:03:00.330211 22156 context-module-impl.cc:244] ContextModule starts to provide model resources: 2021-04-06T11:03:00.33021148+08:00 I0406 11:03:00.330333 22156 context-module-impl.cc:281] ContextModule finished providing model resources : 2021-04-06T11:03:00.330333605+08:00 elapsed: 122.125us I0406 11:03:00.336258 22156 terse_processor.cc:1438] Resetting Terse Processor I0406 11:03:00.336319 22156 terse_processor.cc:941] Cancelling session. W0406 11:03:00.336492 22176 portable_intended_query_stream.cc:235] Exiting due to stream cancellation. W0406 11:03:00.336942 22156 decoder_endpointer_stream.cc:40] Acoustic ep reader thread cancelled. I0406 11:03:00.337181 22156 terse_processor.cc:850] Setup completed I0406 11:03:00.337196 22156 soda_impl.cc:593] Server ASR Disabled I0406 11:03:00.337201 22156 soda_impl.cc:653] Initializing audio logger W0406 11:03:00.337240 22156 soda_async_impl.cc:442] SODA session starting (require_hotword:0, hotword_timeout_in_millis:0, trigger_type:TRIGGER_TYPE_UNSPECIFIED, hybrid_asr_config.mode:MODE_DEFAULT) I0406 11:03:00.337380 22156 soda_async_impl.cc:639] Session parameters updated. Reconfiguring SODA. W0406 11:03:00.337562 22157 soda_async_impl.cc:788] SODA stopped processing audio, mics audio processed in millis: 0, loopback audio processed in millis: 0 I0406 11:03:00.337794 22157 terse_processor.cc:1438] Resetting Terse Processor I0406 11:03:00.337804 22157 terse_processor.cc:941] Cancelling session. W0406 11:03:00.337817 22157 soda_async_impl.cc:840] SODA session stopped due to: STOP_CALLED I0406 11:03:00.337855 22158 recognition_event_delegate.cc:37] Soda session stopped due to: STOP_CALLED W0406 11:03:00.360020 22156 soda_async_impl.cc:911] Deleting soda_impl

benhuang2018 commented 3 years ago

i try to look into the log, and find this message suspicious, any cue / angle i can look into..?

"SODA could not build Hotquery Matcher"

biemster commented 3 years ago

You did everything correct, except you're missing the final step. The soda library contains a couple checks that prevents it from running outside the chrome process; an API key verification, a call stack verification and an unimplemented user verification. You need to disassemble the binary, with Ghidra for example, and path it using a NOP slide or similar to skip those checks.

biemster commented 3 years ago

also you don't need sudo, although that wouldn't fix your problem

benhuang2018 commented 3 years ago

Hi @biemster

First of all, thanks for your kind response. I try to follow some Ghidra tutorial on youtube, and poke around the SODA.dll, it is a whole new thing to me, but i'm stuck in the same place in last few days, so i try to log down some of the things i've done and see maybe if you can share me some insight? I'll remove the post immediately if you find the content is not appropriate to post here..

1 - So, the first step i take is to trace the entry point of the main function in gasr.c and see where it interact with the SODA.dll, from the gasr.c, i believe that the only entry point should be the createSodaAsync function. screenshot_371

2 - Next, i validate it by insert a console out print line after createSodaAsync and rebuild the project, and i observed that the warning message was indeed generated before the debug print line, which convince me that the validation was indeed happened inside the createSodaAsync function. screenshot_374 screenshot_376

3 - Then i load the SODA.dll into Ghidra and decompile it, after search to createSodaAsync function, the first few line till the end was mostly moving data around, except the line in address 180005582, which call the function FUN_180005740 screenshot_380

4 - So i navigate to FUN_180005740, where i found quite a number of checking before triggering the actual operator_new function in address 1800058c7. therefore, i try to change some/all checking function (JZ) to either (NOP) or (JNZ), but it doesn't seems quite effective... screenshot_383 screenshot_385

Am I even looking into the right place? Or did i simply just not understand what FUN_180005740 is doing...?

biemster commented 3 years ago

You are at exactly the right spot, well done! The final result of those three if-statements combined is assigned to a struct variable a couple lines below this codeblock, except if one of them did not pass. You'll need to be a bit creative there (I for example made the first JZ jump just a couple bytes further to skip the resetting of the final result, but there are other ways like NOPs indeed). After that take care that that particular struct variable is also initialized to "true" (0xFF, anything except 0x0) a couple lines above this code block, as it is now initialized to "false". This should get you a properly patched DLL, and don't forget to do the same for the CreateExtendedSodaAsync function if you plan to use the python client.

benhuang2018 commented 3 years ago

Hi @biemster

I've been trying to implement it, but unfortunately things didn't go so well...

below basically is the 2 things i've done:

1 - bypassing checking from my understanding, the first 3 if checking here should be most important, and they generate a local variable cVar1 and uVar2, which generate a pattern and then apply masking for value checking done by instruction "pmovmskb" for vairable iVar3 screenshot_392

2 - struct variable only after checking with masked variable generate iVar3 with pmovmskb, another checking will be apply also with instruction "pmovmskb", and define the value of bVar5, which seems to be the only variable affecting the overall flow before entering the function operator_new at the end of function. screenshot_398

So, I take many approach to try to bypass these logic, the most naïve, however, should also be the most effective one is, i change all the instruction for the whole if else checking to NOP, and set the AL to 1 and assign into the variable screenshot_400

but the output seems the same... what did i missed... screenshot_403

biemster commented 3 years ago

Your disassembly looks a bit different from mine (since I did it on a much earlier version), so could you try the following (not sure if it is going to work): your (local_40 + 4) seems to be where the result of the checks is stored. On line 66 it is initialized to "false", you should change that to "true". Then you need to find a way to skip line 182, either by NOPping those instructions, or by having the first JZ jump a couple bytes further, so it is skipped. Third, you are running the executable without any input audio! You need to pipe the mic output into it, or a wav file. Take care they have the correct bitrate and such.

benhuang2018 commented 3 years ago

sadly it doesn't work...

screenshot_407

screenshot_405

I followed the steps but the output is still the same... I run it on ubuntu with samples_cpp_windows_console_samples_whatstheweatherlike.wav, but not work... screenshot_409

The soda version i got is 1.0.2, since there is no way to acquire the old version anymore, any chance you can email me the version you got..?

biemster commented 3 years ago

you got a mistake in your command on linux, could you try

wine gasr.exe --stream-delay < samples_cpp_windows_console_samples_whatstheweatherlike.wav
BoboBananas commented 3 years ago

Hi, I'm also working on this and have done the bypass for the checking and am getting the following output:

W0415 11:51:30.542577 1 soda_async_impl.cc:260] Creating soda_impl Segmentation fault: 11

This is using the python version

biemster commented 3 years ago

@BoboBananas that looks like your patch went wrong, but this is too little info to make any educated guess. Please try some different ways to bypass the checks and see what works.

benhuang2018 commented 3 years ago

Hi @biemster

Finally I got it working, you are right! i run the line wine gasr.exe --stream-delay < samples_cpp_windows_console_samples_whatstheweatherlike.wav and the result show as expected. Thanks so much! probably it was working all the time and i was getting this invalid message just because i test it wrongly...@@

anyway, now i'm testing everything with this 2 commands, work for windows just remove the wine command

ffmpeg -re -i samples_cpp_windows_console_samples_whatstheweatherlike.wav -ac 1 -ar 16000 -acodec pcm_s16le -f s16le pipe:1 | wine gasr.exe --stream-delay

ffmpeg -re --i audio="@devicecm{33D9A762-90C8-11D0-BD43-00A0C911CE86}\wave_{311E49D3-66E8-4BD1-B0D2-9963D9136428}" -ac 1 -ar 16000 -acodec pcm_s16le -f s16le pipe:1 | wine gasr.exe --stream-delay

Now I'm moving to testing on different language from dl.google.com, as far as i tested, english is definitely working, and Spanish as well,

but sadly for asian language, ja-JP, cmn-Hant-TW, cmn-Hans-CN throw error, screenshot_414

probably because current version of libsoda were not fully designed to be fully support all language..? Anyway, i got the offline speech recognition running on a old android 4.4 device on GBoard, so i'm looking into it following below link https://github.com/AIFanatic/google-offline-speech-recognition

harshgsx commented 3 years ago

@benhuang2018 Did you add the API key?

benhuang2018 commented 3 years ago

You did everything correct, except you're missing the final step. The soda library contains a couple checks that prevents it from running outside the chrome process; an API key verification, a call stack verification and an unimplemented user verification. You need to disassemble the binary, with Ghidra for example, and path it using a NOP slide or similar to skip those checks.

Hi @harshshah903 Like biemster said, you need to bypass the logic, so no, you dont need the api key But again, i think it is important to mention again from biemster's blog post, it is not offically authorized, so just use this for learning/studying

cppxaxa commented 2 years ago

Hi @biemster, After reading this issue post, and encountering the new API - CreateExtendedSodaAsync in the Chrome version - Version 98.0.4758.102 (Official Build) (64-bit) SODA version 1.1.0 Python script modification repo - https://github.com/cppxaxa/gasr-py-001

Thanks for taking time to read further !

I came across 2 doubts -

  1. Should I open a new issue or continue in the same?
  2. Technical doubt on the following - I think some part of the function "FUN_180004220" is the validation check. (I'm new to this and it seems complicated for me :( )

Can you please guide me further ? I really want to see this working, but unable to figure out any matching logic as posted in this github issue.

I started off with the following and observed the two function calls EntryFunc EntryFunc_2

FUN_181494ca0 FUN_181494ca0.txt

FUN_180004220 FUN_180004220.txt

Other reference function that I tried to understand, but seems too tricky for me

FUN_1800104b0.txt FUN_180004370.txt FUN_180004390.txt FUN_1814efec2.txt FUN_1814f0088.txt FUN_18145cac0.txt FUN_18145d720.txt FUN_18147d830.txt

biemster commented 2 years ago

@cppxaxa you should disassemble either the linux or osx library as well, they still have the debug symbols in them. Then your FUN_*** function names will ALL be replaced with the actual function names in the code, not only the exported ones. This makes it much easier to patch, since they are produced from the same code base. Just find the function to patch in the linux/osx lib, and patch the win dll in that same location.

cppxaxa commented 2 years ago

@biemster, First of all, I switched to Ubuntu 18.04 and got hold of same Chrome version as my win, and copied the "libsoda.so" After spending some time in exploring the function "CreateExtendedSodaAsync" (3 hours) using Ghidra, I felt as if there's no validation present in the function.

Then I tried debugging the python file and it seems the "CreateExtendedSodaAsync" is able to give out an instance. But the "ExtendedSodaStart" gives the Segmentation fault.

My doubt - Should I start to look into the ExtendedSodaStart instead of CreateExtendedSodaAsync to look for validations like CONCAT (as posted in this github issue) ? or am I missing something ?

Reference images - Debugging session of python image

Value of self.handle in debugging session image

Just a run with stdout and stderr messages in terminal - image

biemster commented 2 years ago

The CreateExtendedSodaAsync is the start of the code path you need to go down.

cppxaxa commented 2 years ago

An update from my side, the "libsoda.so" is really readable ! <-- Thanks for the hint

CreateExtendedSodaAsync image

Some place under CreateExtendedSodaAsync, where I suspect a validation. I will try to shortlist some places first myself to bother you lesser image