I've mentioned this on flashlight repo but haven't heard anything, wondered if I might get a response here?
I have installed flashlight asr and want to do some fine tuning on the RASR model. I have downloaded the Transformer 70M model and prepared my own dataset of an hour of audio. I am following the jupyter notebook "FinetuneCTC.ipynb".
I have built using vcpgk, then via docker, and now just finished building from source, but still having issues running fine tuning. I can run fl_asr_test perfectly fine on the same training set, but fl_asr_tutorial_finetune_ctc crashes as per below exception.
There are 373 items in train.lst and 38 items in valid.lst.
Platform and Hardware
- Ubuntu 18.04
- NVIDIA Telsa K80
- Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz
- CUDA 11.4 with 470.103.01 driver.
- 50GB of RAM
./fl_asr_test --am /media/ubuntu/1tb/models/am/1-transformer-70M/am_transformer_ctc_stride3_letters_70Mparams.bin \
> --datadir /media/ubuntu/1tb/datasets/lara/ --emission_dir '' \
> --uselexicon false \
> --test test.lst --tokens /media/ubuntu/1tb/models/lm/tokens.txt --lexicon /media/ubuntu/1tb/models/lm/lexicon.txt --show
I0202 17:15:53.568406 52064 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
I0202 17:15:57.607283 6476 Test.cpp:111] Gflags after parsing
--flagfile=; --fromenv=; --tryfromenv=; --undefok=; --tab_completion_columns=80; --tab_completion_word=; --help=false; --helpfull=false; --helpmatch=; --helpon=; --helppackage=false; --helpshort=false; --helpxml=false; --version=false; --adambeta1=0.94999999999999996; --adambeta2=0.98999999999999999; --am=/media/ubuntu/1tb/models/am/1-transformer-70M/am_transformer_ctc_stride3_letters_70Mparams.bin; --am_decoder_tr_dropout=0.20000000000000001; --am_decoder_tr_layerdrop=0.20000000000000001; --am_decoder_tr_layers=6; --arch=EG_GLU1x2048_S3_TR36x384_1536_DO0.05_LD0.05_CTC; --attention=keyvalue; --attentionthreshold=2147483647; --attnWindow=softPretrain; --attnconvchannel=0; --attnconvkernel=0; --attndim=0; --batching_max_duration=0; --batching_strategy=none; --batchsize=4; --beamsize=2500; --beamsizetoken=250000; --beamthreshold=25; --channels=1; --criterion=ctc; --critoptim=adagrad; --datadir=/media/ubuntu/1tb/datasets/lara/; --decoderattnround=1; --decoderdropout=0; --decoderrnnlayer=1; --decodertype=wrd; --devwin=0; --emission_dir=; --emission_queue_size=3000; --enable_distributed=true; --encoderdim=256; --eosscore=0; --everstoredb=false; --features_type=mfsc; --fftcachesize=1; --filterbanks=80; --fl_amp_max_scale_factor=32000; --fl_amp_scale_factor=4096; --fl_amp_scale_factor_update_interval=2000; --fl_amp_use_mixed_precision=false; --fl_benchmark_mode=true; --fl_log_level=; --fl_log_mem_ops_interval=0; --fl_optim_mode=; --fl_vlog_level=0; --flagsfile=; --framesizems=25; --framestridems=10; --gamma=1; --gumbeltemperature=1; --highfreqfilterbank=-1; --inputfeeding=false; --isbeamdump=false; --iter=9223372036854775807; --itersave=false; --labelsmooth=0.050000000000000003; --leftWindowSize=50; --lexicon=/media/ubuntu/1tb/models/lm/lexicon.txt; --linlr=-1; --linlrcrit=-1; --linseg=0; --lm=; --lm_memory=5000; --lm_vocab=; --lmtype=kenlm; --lmweight=0; --lmweight_high=4; --lmweight_low=0; --lmweight_step=0.20000000000000001; --localnrmlleftctx=0; --localnrmlrightctx=0; --logadd=false; --lowfreqfilterbank=0; --lr=0.02; --lr_decay=100; --lr_decay_step=40; --lrcosine=false; --lrcrit=0.02; --max_devices_per_node=8; --maxdecoderoutputlen=400; --maxgradnorm=0.10000000000000001; --maxload=-1; --maxrate=10; --maxsil=50; --maxword=-1; --melfloor=1; --mfcccoeffs=13; --minrate=3; --minsil=0; --momentum=0.94999999999999996; --netoptim=adagrad; --nthread=6; --nthread_decoder=1; --nthread_decoder_am_forward=1; --numattnhead=8; --onorm=target; --optimepsilon=1e-08; --optimrho=0.90000000000000002; --pctteacherforcing=99; --pcttraineval=1; --pretrainWindow=0; --replabel=0; --reportiters=4000; --rightWindowSize=50; --rndv_filepath=; --rundir=/checkpoint/padentomasello/wav2letter_experiments/256_GPU; --samplerate=16000; --sampletarget=0.01; --samplingstrategy=rand; --saug_fmaskf=30; --saug_fmaskn=2; --saug_start_update=24000; --saug_tmaskn=10; --saug_tmaskp=0.050000000000000003; --saug_tmaskt=30; --sclite=; --seed=0; --sfx_config=; --sfx_start_update=2147483647; --show=true; --showletters=false; --silscore=0; --smearing=none; --smoothingtemperature=1; --softwoffset=10; --softwrate=5; --softwstd=4; --sqnorm=true; --stepsize=9223372036854775807; --surround=; --test=test.lst; --tokens=/media/ubuntu/1tb/models/lm/tokens.txt; --train=/private/home/qiantong/wav2letter_experiments/256_GPU/join_train_notl_maxlen.25s.lst; --trainWithWindow=true; --transdiag=0; --unkscore=-inf; --use_memcache=false; --uselexicon=true; --usewordpiece=false; --valid=dev-clean:/checkpoint/jacobkahn/data/lists/librispeech/dev-clean.lst,dev-other:/checkpoint/jacobkahn/data/lists/librispeech/dev-other.lst,test-clean:/checkpoint/jacobkahn/data/lists/librispeech/test-clean.lst,test-other:/checkpoint/jacobkahn/data/lists/librispeech/test-other.lst,hub05_callhome:/checkpoint/wav2letter/data/swbd_lists/noNL/hub05-callhome.16khz.lst,hub05_switchboard:/checkpoint/wav2letter/data/swbd_lists/noNL/hub05-switchboard.16khz.lst,commonvoice_dev:/checkpoint/wav2letter/data/commonvoice/lists/dev.lst,commonvoice_test:/checkpoint/wav2letter/data/commonvoice/lists/test.lst,nov92:/checkpoint/qiantong/wsj/nov92.lst; --validbatchsize=-1; --warmup=48000; --weightdecay=0; --wordscore=0; --wordseparator=|; --world_rank=0; --world_size=1; --alsologtoemail=; --alsologtostderr=false; --colorlogtostderr=false; --drop_log_memory=true; --log_backtrace_at=; --log_dir=; --log_link=; --log_prefix=true; --logbuflevel=0; --logbufsecs=30; --logemaillevel=999; --logfile_mode=436; --logmailer=/bin/mail; --logtostderr=true; --max_log_size=1800; --minloglevel=0; --stderrthreshold=2; --stop_logging_if_full_disk=false; --symbolize_stacktrace=true; --v=0; --vmodule=;
I0202 17:15:57.608184 6476 Test.cpp:136] Number of classes (network): 29
I0202 17:15:57.887734 6476 Test.cpp:143] Number of words: 200001
I0202 17:15:58.247447 6476 Test.cpp:205] [Dataset] Dataset loaded, with 10 samples.
|T|: a h | t h i s | f i r e | h a s | b e e n | e x t i n g u i s h e d | w i t h | a | l i n e | o f | t h i r t y | e i g h t | a h | t h e | p o w e r | h a s | b e e n | i s o l a t e d | b y | t h e | e n e r g y | a u t h o r i t y | u m | w e | h a v e | c r e w s | a t | w o r k | j u s t | d o i n g | s a l v a g e | a n d | o v e r h a u l | a h | w e | c a n | m a k e | o u r | t a n k e r | a v a i l a b l e | t o | r e s p o n d | u m | i n | a d d i t i o n | t o | t h a t | o n e | m a l e | u m | h a s | b e e n | t r a n s p o r t e d | b y | a m b u l a n c e | a h | f o r | u m | m i n o r | b u r n s | a n d | s m o k e | i n h a l a t i o n | o v e r
|P|: h a v e | i t f a r a s | b e e n | e x t i n g u i s h e d | w i t h | l i n e | o r | t h i r t y | e i g h t | u h | t h e | p o w e r | h a s | b e e n | i s o l a t e d | b y | t h e | e n e r g y | a u t h o r i t y | u m | w e | h a v e | c r u i s | a t | w o r k | j u s t | d o i n g | s a l v a g e | a n d | a r i v e r h a u l | u h | w e | c a n | m a k e | o u r | t a n k e r | a v a i l a b l e | t o | r e s p o n d | u m | i n | a d d i t i o n | t o | t h a t | o n e | m a l e | u m | h a s | b e e n | t r a n s p o r t e d | b y | a m b u l a n c e | u h | f o r | u m | m o n o | b u r n s | a n d | s m o k i n g | e l a t i o n | o v e r
[sample: 280, WER: 21.875%, TER: 8.57988%, total WER: 21.875%, total TER: 8.57988%, progress (thread 0): 10%]
|T|: f i r e | c o m m s | t h a n k | y o u | p u m p e r | t h r e e | s i x | t h r e e | a h | r e d u n d a n t | m a c h i n e r y | a p p e a r s | t o | b e | a | r o c k | t u m b l e r | a n d | t y r e s | t h a t | c a u g h t | o n | f i r e | a h | t h e | f i r e | i s | n o w | o u t | y o u | a r e | p r o v i d i n g | r f s | w a t e r | t o | c o o l | t h e | m a c h i n e r y | n o | f u r t h e r | n e e d s | a t | t h i r t e e n | t e n | c h a n n e l | c l e a r
|P|: ' s b i c k a r m s | t h i n k | e | p u m p a | t h r e e | s i x | t h r e e | u h | r e d u n d a n t | m a c h i n e r y | r e p u r s | t o | b e | a | r o c k | t u m b l e r | a n d | i t ' s | t i e s | a t | c a u g h t | o n | f i r e | u h | t h e | f i r e i s | n o w | o u t | y o u ' r e | p r o v i d i n g | i r | f e c e | w a t e r | t o | c a l l | t h e | m a c h i n e r y | m o c | f o r | t h e | n e e d s | a t | t h i r t y | e i n h | t e n | t e n | o ' c l e c k
[sample: 240, WER: 54.3478%, TER: 22.8571%, total WER: 35.4545%, total TER: 14.5798%, progress (thread 0): 20%]
|T|: p u m p | t w e n t y | s i x | g r e e n | c a r s | b e i n g | m a d e | s a f e | w i n d o w s | b e i n g | t a p e d | u p | a n d | l e f t | w i t h | o w n e r | w e ' r e | c o d e | | o v e r
|P|: s o ' m | t w e n t y | s i x | g r e e n | c a r s | b e i n g | m a d e | s a f e | w i n d o w s | b e i n g | t a p e d | u p | a n d | l e f t | w i t h | o w n e r | w a r | c o p e | f o r e o v e r
[sample: 300, WER: 25%, TER: 11.7647%, total WER: 33.8462%, total TER: 14.1606%, progress (thread 0): 30%]
|T|: a n d | r e m e m b e r | h e ' s | f r o m | a h | f i v e | b r a v o | t h e | c o v i d | w a r d
|P|: u h | a n d | r e m e m b r h e s | f r o m | u h | f i v e | b r i v a | t h e | c a v e r i t | w a r t
[sample: 54, WER: 70%, TER: 27.451%, total WER: 36.4286%, total TER: 15.0815%, progress (thread 0): 40%]
|T|: s y d n e y | t o | d o u b l e | t h r e e | o n e | c o u l d | y o u | p l e a s e | c o n t a c t | t h e | n o r t h | b o a r d | f o r | a | b
|P|: t e n t y | t w o | d o u b l e | t h r e e | o n e | ' c o u l d | y o u | p l e a s e | c o n t a c t | t h e | n o r t h b o a r d | f o r | a | o n e | b
[sample: 309, WER: 46.6667%, TER: 14.6667%, total WER: 37.4194%, total TER: 15.0432%, progress (thread 0): 50%]
|T|: y e a h | t h a t ' s | f i n e | a h | i f | a h | h e | g o e s | d i r e c t | t o | g e o r g e ' s | r i v e r | t h a t | i ' l l | b e | g r e a t
|P|: t h a t ' s | f i n e | u h | i f | u h | h e | g o e s | d i r e c t | t h o | g e o r g i ' s | r i v e r | t h e ' l l | b e | g r e a t
[sample: 65, WER: 43.75%, TER: 16.8831%, total WER: 38.0117%, total TER: 15.2027%, progress (thread 0): 60%]
|T|: a h | s o t | f o r t y | t w o | t h a t | w o u l d | b e | f a n t a s t i c | i f | i | c o u l d | g e t | y o u | s t a r t e d | i n | t h a t | d i r e c t i o n
|P|: a h | t h o u g h t | f o r t y | t w o | t h a t ' d | b e | u h | f a n t a s t i c | i f | i | c o u l d | g e t | y o u | s a r e d | i n | t h a t | d i r e c t i o n
[sample: 27, WER: 29.4118%, TER: 17.6471%, total WER: 37.234%, total TER: 15.4162%, progress (thread 0): 70%]
|T|: s o u t h e r n | o p s | t o | o a k | f l a t s | n i n e | t w e l v e | f o r | a | h o s p i t a l | u p d a t e
|P|: c o m l i n g | u p | s t o k e s | h a t ' s | n i n e | t w e l v e | f o r | a | h o s p i t a l | u p d a t e
[sample: 87, WER: 45.4545%, TER: 28.8136%, total WER: 37.6884%, total TER: 16.1822%, progress (thread 0): 80%]
|T|: f i r e | c o m m | n o r t h e r n | b e a c h e s | d u t y
|P|: h i | c a l m | n o r t h e r n | b e a c h | c u t y
[sample: 249, WER: 80%, TER: 25.8065%, total WER: 38.7255%, total TER: 16.4628%, progress (thread 0): 90%]
|T|: t h a n k s | s o | m u c h
|P|: t h i n k | s o | m u c h
[sample: 44, WER: 33.3333%, TER: 14.2857%, total WER: 38.6473%, total TER: 16.4345%, progress (thread 0): 100%]
I've seen other issues raised similar to this where it was requested that the person who raised the issue run the failed tests individually and return the output, so here it is.
40: Test command: /root/flashlight/build/flashlight/fl/test/AutogradTest "--gtest_filter=AutogradTest.AfRefCountBasic"
40: Test timeout computed to be: 1500
40: I0202 18:52:40.248621 77600 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
40: Note: Google Test filter = AutogradTest.AfRefCountBasic
40: [==========] Running 1 test from 1 test suite.
40: [----------] Global test environment set-up.
40: [----------] 1 test from AutogradTest
40: [ RUN ] AutogradTest.AfRefCountBasic
40: /root/flashlight/flashlight/fl/test/autograd/AutogradTest.cpp:79: Failure
40: Expected equality of these values:
40: refCount
40: Which is: 0
40: 1
40: [ FAILED ] AutogradTest.AfRefCountBasic (0 ms)
40: [----------] 1 test from AutogradTest (0 ms total)
40:
40: [----------] Global test environment tear-down
40: [==========] 1 test from 1 test suite ran. (0 ms total)
40: [ PASSED ] 0 tests.
40: [ FAILED ] 1 test, listed below:
40: [ FAILED ] AutogradTest.AfRefCountBasic
40:
40: 1 FAILED TEST
1/1 Test #40: AutogradTest.AfRefCountBasic .....***Failed 0.11 sec
41: Test command: /root/flashlight/build/flashlight/fl/test/AutogradTest "--gtest_filter=AutogradTest.AfRefCountModify"
41: Test timeout computed to be: 1500
41: I0202 18:53:38.692444 22496 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
41: Note: Google Test filter = AutogradTest.AfRefCountModify
41: [==========] Running 1 test from 1 test suite.
41: [----------] Global test environment set-up.
41: [----------] 1 test from AutogradTest
41: [ RUN ] AutogradTest.AfRefCountModify
41: /root/flashlight/flashlight/fl/test/autograd/AutogradTest.cpp:101: Failure
41: Expected equality of these values:
41: refCount
41: Which is: 0
41: 1
41: [ FAILED ] AutogradTest.AfRefCountModify (0 ms)
41: [----------] 1 test from AutogradTest (0 ms total)
41:
41: [----------] Global test environment tear-down
41: [==========] 1 test from 1 test suite ran. (0 ms total)
41: [ PASSED ] 0 tests.
41: [ FAILED ] 1 test, listed below:
41: [ FAILED ] AutogradTest.AfRefCountModify
41:
41: 1 FAILED TEST
42: Test command: /root/flashlight/build/flashlight/fl/test/AutogradTest "--gtest_filter=AutogradTest.AfRefCountGradient"
42: Test timeout computed to be: 1500
42: I0202 18:53:58.560300 12640 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
42: Note: Google Test filter = AutogradTest.AfRefCountGradient
42: [==========] Running 1 test from 1 test suite.
42: [----------] Global test environment set-up.
42: [----------] 1 test from AutogradTest
42: [ RUN ] AutogradTest.AfRefCountGradient
42: /root/flashlight/flashlight/fl/test/autograd/AutogradTest.cpp:139: Failure
42: Expected equality of these values:
42: refCount
42: Which is: 0
42: 1
42: [ FAILED ] AutogradTest.AfRefCountGradient (0 ms)
42: [----------] 1 test from AutogradTest (0 ms total)
42:
42: [----------] Global test environment tear-down
42: [==========] 1 test from 1 test suite ran. (0 ms total)
42: [ PASSED ] 0 tests.
42: [ FAILED ] 1 test, listed below:
42: [ FAILED ] AutogradTest.AfRefCountGradient
42:
42: 1 FAILED TEST
116: Test command: /root/flashlight/build/flashlight/fl/test/AutogradTest "--gtest_filter=AutogradTest.GetAdvancedIndex"
116: Test timeout computed to be: 1500
116: I0202 18:54:52.088248 22208 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
116: Note: Google Test filter = AutogradTest.GetAdvancedIndex
116: [==========] Running 1 test from 1 test suite.
116: [----------] Global test environment set-up.
116: [----------] 1 test from AutogradTest
116: [ RUN ] AutogradTest.GetAdvancedIndex
116: unknown file: Failure
116: C++ exception with description "[/root/flashlight/flashlight/fl/autograd/backend/cuda/operators/AdvancedIndex.cu:184] CUDA error: no kernel image is available for execution on the device" thrown in the test body.
116: [ FAILED ] AutogradTest.GetAdvancedIndex (536 ms)
116: [----------] 1 test from AutogradTest (536 ms total)
116:
116: [----------] Global test environment tear-down
116: [==========] 1 test from 1 test suite ran. (536 ms total)
116: [ PASSED ] 0 tests.
116: [ FAILED ] 1 test, listed below:
116: [ FAILED ] AutogradTest.GetAdvancedIndex
116:
116: 1 FAILED TEST
1/2 Test #116: AutogradTest.GetAdvancedIndex ......***Failed 0.69 sec
336: Test command: /root/flashlight/build/flashlight/fl/test/ArrayFireTensorBaseTest "--gtest_filter=ArrayFireTensorBaseTest.AfRefCountBasic"
336: Test timeout computed to be: 1500
336: Running main() from /root/googletest/googletest/src/gtest_main.cc
336: Note: Google Test filter = ArrayFireTensorBaseTest.AfRefCountBasic
336: [==========] Running 1 test from 1 test suite.
336: [----------] Global test environment set-up.
336: [----------] 1 test from ArrayFireTensorBaseTest
336: [ RUN ] ArrayFireTensorBaseTest.AfRefCountBasic
336: /root/flashlight/flashlight/fl/test/tensor/ArrayFireTensorBaseTest.cpp:78: Failure
336: Expected equality of these values:
336: refCount
336: Which is: 0
336: 1
336: [ FAILED ] ArrayFireTensorBaseTest.AfRefCountBasic (52 ms)
336: [----------] 1 test from ArrayFireTensorBaseTest (52 ms total)
336:
336: [----------] Global test environment tear-down
336: [==========] 1 test from 1 test suite ran. (52 ms total)
336: [ PASSED ] 0 tests.
336: [ FAILED ] 1 test, listed below:
336: [ FAILED ] ArrayFireTensorBaseTest.AfRefCountBasic
336:
336: 1 FAILED TEST
1/1 Test #336: ArrayFireTensorBaseTest.AfRefCountBasic ...***Failed 0.11 sec
339: Test command: /root/flashlight/build/flashlight/fl/test/ArrayFireTensorBaseTest "--gtest_filter=ArrayFireTensorBaseTest.ArrayFireAssignmentOperators"
339: Test timeout computed to be: 1500
339: Running main() from /root/googletest/googletest/src/gtest_main.cc
339: Note: Google Test filter = ArrayFireTensorBaseTest.ArrayFireAssignmentOperators
339: [==========] Running 1 test from 1 test suite.
339: [----------] Global test environment set-up.
339: [----------] 1 test from ArrayFireTensorBaseTest
339: [ RUN ] ArrayFireTensorBaseTest.ArrayFireAssignmentOperators
339: /root/flashlight/flashlight/fl/test/tensor/ArrayFireTensorBaseTest.cpp:118: Failure
339: Expected equality of these values:
339: refCount
339: Which is: 0
339: 1
339: [ FAILED ] ArrayFireTensorBaseTest.ArrayFireAssignmentOperators (51 ms)
339: [----------] 1 test from ArrayFireTensorBaseTest (51 ms total)
339:
339: [----------] Global test environment tear-down
339: [==========] 1 test from 1 test suite ran. (51 ms total)
339: [ PASSED ] 0 tests.
339: [ FAILED ] 1 test, listed below:
339: [ FAILED ] ArrayFireTensorBaseTest.ArrayFireAssignmentOperators
339:
339: 1 FAILED TEST
test 372
Start 372: CriterionTest.CTCEmptyTarget
372: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.CTCEmptyTarget"
372: Test timeout computed to be: 1500
372: I0202 18:57:10.772891 71488 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
372: Note: Google Test filter = CriterionTest.CTCEmptyTarget
372: [==========] Running 1 test from 1 test suite.
372: [----------] Global test environment set-up.
372: [----------] 1 test from CriterionTest
372: [ RUN ] CriterionTest.CTCEmptyTarget
372: unknown file: Failure
372: C++ exception with description "Error: compute_ctc_loss, stat = execution failed" thrown in the test body.
372: [ FAILED ] CriterionTest.CTCEmptyTarget (535 ms)
372: [----------] 1 test from CriterionTest (535 ms total)
372:
372: [----------] Global test environment tear-down
372: [==========] 1 test from 1 test suite ran. (535 ms total)
372: [ PASSED ] 0 tests.
372: [ FAILED ] 1 test, listed below:
372: [ FAILED ] CriterionTest.CTCEmptyTarget
372:
372: 1 FAILED TEST
1/23 Test #372: CriterionTest.CTCEmptyTarget ..................................***Failed 0.69 sec
test 373
Start 373: CriterionTest.CTCCost
373: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.CTCCost"
373: Test timeout computed to be: 1500
373: I0202 18:57:11.443037 00032 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
373: Note: Google Test filter = CriterionTest.CTCCost
373: [==========] Running 1 test from 1 test suite.
373: [----------] Global test environment set-up.
373: [----------] 1 test from CriterionTest
373: [ RUN ] CriterionTest.CTCCost
373: unknown file: Failure
373: C++ exception with description "Error: compute_ctc_loss, stat = execution failed" thrown in the test body.
373: [ FAILED ] CriterionTest.CTCCost (525 ms)
373: [----------] 1 test from CriterionTest (525 ms total)
373:
373: [----------] Global test environment tear-down
373: [==========] 1 test from 1 test suite ran. (525 ms total)
373: [ PASSED ] 0 tests.
373: [ FAILED ] 1 test, listed below:
373: [ FAILED ] CriterionTest.CTCCost
373:
373: 1 FAILED TEST
2/23 Test #373: CriterionTest.CTCCost .........................................***Failed 0.66 sec
test 374
Start 374: CriterionTest.CTCJacobian
374: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.CTCJacobian"
374: Test timeout computed to be: 1500
374: I0202 18:57:12.104254 56832 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
374: Note: Google Test filter = CriterionTest.CTCJacobian
374: [==========] Running 1 test from 1 test suite.
374: [----------] Global test environment set-up.
374: [----------] 1 test from CriterionTest
374: [ RUN ] CriterionTest.CTCJacobian
374: Failed to allocate memory of size 16777215.36 TiB (Device: 1, Capacity: 11.17 GiB, Allocated: 2.00 MiB, Cached: 1.97 MiB) with error 'ArrayFire Exception (Device out of memory:101):
374: ArrayFire error:
374: In function fl::MemoryManagerInstaller::MemoryManagerInstaller(std::shared_ptr<fl::MemoryManagerAdapter>)::<lambda(size_t)>
374: In file /root/flashlight/flashlight/fl/memory/MemoryManagerInstaller.cpp:180'
374: unknown file: Failure
374: C++ exception with description "ArrayFire Exception (Device out of memory:101):
374: In function alloc
374: In file src/api/c/memory.cpp:738
374:
374:
374: 0# 0x00007FE0B1AB691D in /usr/lib/libafcuda.so.3
374: 1# 0x00007FE0B118A376 in /usr/lib/libafcuda.so.3
374: 2# 0x00007FE0B0BA071D in /usr/lib/libafcuda.so.3
374: 3# 0x00007FE0B0BA07E9 in /usr/lib/libafcuda.so.3
374: 4# 0x00007FE0B191DFD4 in /usr/lib/libafcuda.so.3
374: 5# af_create_handle in /usr/lib/libafcuda.so.3
374: 6# 0x00007FE0B1C4025C in /usr/lib/libafcuda.so.3
374: 7# af::array::array(long long, af_dtype) in /usr/lib/libafcuda.so.3
374: 8# fl::pkg::speech::ConnectionistTemporalClassificationCriterion::forward(std::vector<fl::Variable, std::allocator<fl::Variable> > const&) in /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest
374: 9# 0x00005575A67C9835 in /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest
374: 10# 0x00005575A67CF90A in /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest
374: 11# CriterionTest_CTCJacobian_Test::TestBody() in /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest
374: 12#" thrown in the test body.
374: [ FAILED ] CriterionTest.CTCJacobian (566 ms)
374: [----------] 1 test from CriterionTest (566 ms total)
374:
374: [----------] Global test environment tear-down
374: [==========] 1 test from 1 test suite ran. (566 ms total)
374: [ PASSED ] 0 tests.
374: [ FAILED ] 1 test, listed below:
374: [ FAILED ] CriterionTest.CTCJacobian
374:
374: 1 FAILED TEST
3/23 Test #374: CriterionTest.CTCJacobian .....................................***Failed 0.70 sec
test 375
Start 375: CriterionTest.Batching
375: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.Batching"
375: Test timeout computed to be: 1500
375: I0202 18:57:12.807091 28960 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
375: Note: Google Test filter = CriterionTest.Batching
375: [==========] Running 1 test from 1 test suite.
375: [----------] Global test environment set-up.
375: [----------] 1 test from CriterionTest
375: [ RUN ] CriterionTest.Batching
375: Failed to allocate memory of size 16777214.97 TiB (Device: 1, Capacity: 11.17 GiB, Allocated: 2.00 MiB, Cached: 1.98 MiB) with error 'ArrayFire Exception (Device out of memory:101):
375: ArrayFire error:
375: In function fl::MemoryManagerInstaller::MemoryManagerInstaller(std::shared_ptr<fl::MemoryManagerAdapter>)::<lambda(size_t)>
375: In file /root/flashlight/flashlight/fl/memory/MemoryManagerInstaller.cpp:180'
375: unknown file: Failure
375: C++ exception with description "ArrayFire Exception (Device out of memory:101):
375: In function alloc
375: In file src/api/c/memory.cpp:738
375:
375:
375: 0# 0x00007F48AA18091D in /usr/lib/libafcuda.so.3
375: 1# 0x00007F48A9854376 in /usr/lib/libafcuda.so.3
375: 2# 0x00007F48A926A71D in /usr/lib/libafcuda.so.3
375: 3# 0x00007F48A926A7E9 in /usr/lib/libafcuda.so.3
375: 4# 0x00007F48A9FE7FD4 in /usr/lib/libafcuda.so.3
375: 5# af_create_handle in /usr/lib/libafcuda.so.3
375: 6# 0x00007F48AA30A25C in /usr/lib/libafcuda.so.3
375: 7# af::array::array(long long, af_dtype) in /usr/lib/libafcuda.so.3
375: 8# fl::pkg::speech::ConnectionistTemporalClassificationCriterion::forward(std::vector<fl::Variable, std::allocator<fl::Variable> > const&) in /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest
375: 9# 0x000055A58360B995 in /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest
375: 10# 0x000055A58361190A in /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest
375: 11# CriterionTest_Batching_Test::TestBody() in /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest
375: 12# vo" thrown in the test body.
375: [ FAILED ] CriterionTest.Batching (571 ms)
375: [----------] 1 test from CriterionTest (571 ms total)
375:
375: [----------] Global test environment tear-down
375: [==========] 1 test from 1 test suite ran. (571 ms total)
375: [ PASSED ] 0 tests.
375: [ FAILED ] 1 test, listed below:
375: [ FAILED ] CriterionTest.Batching
375:
375: 1 FAILED TEST
4/23 Test #375: CriterionTest.Batching ........................................***Failed 0.71 sec
test 376
Start 376: CriterionTest.CTCCompareTensorflow
376: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.CTCCompareTensorflow"
376: Test timeout computed to be: 1500
376: I0202 18:57:13.515019 06144 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
376: Note: Google Test filter = CriterionTest.CTCCompareTensorflow
376: [==========] Running 1 test from 1 test suite.
376: [----------] Global test environment set-up.
376: [----------] 1 test from CriterionTest
376: [ RUN ] CriterionTest.CTCCompareTensorflow
376: unknown file: Failure
376: C++ exception with description "Error: compute_ctc_loss, stat = execution failed" thrown in the test body.
376: [ FAILED ] CriterionTest.CTCCompareTensorflow (526 ms)
376: [----------] 1 test from CriterionTest (526 ms total)
376:
376: [----------] Global test environment tear-down
376: [==========] 1 test from 1 test suite ran. (526 ms total)
376: [ PASSED ] 0 tests.
376: [ FAILED ] 1 test, listed below:
376: [ FAILED ] CriterionTest.CTCCompareTensorflow
376:
376: 1 FAILED TEST
5/23 Test #376: CriterionTest.CTCCompareTensorflow ............................***Failed 0.66 sec
test 377
Start 377: CriterionTest.ViterbiPath
377: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.ViterbiPath"
377: Test timeout computed to be: 1500
377: I0202 18:57:14.176560 29440 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
377: Note: Google Test filter = CriterionTest.ViterbiPath
377: [==========] Running 1 test from 1 test suite.
377: [----------] Global test environment set-up.
377: [----------] 1 test from CriterionTest
377: [ RUN ] CriterionTest.ViterbiPath
377: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
377: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 1.06535e+09 vs 1e-05
377: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
377: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 2 vs 1e-05
377: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
377: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 4.29497e+09 vs 1e-05
377: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
377: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 4.29497e+09 vs 1e-05
377: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
377: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 4.29497e+09 vs 1e-05
377: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
377: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 4.29497e+09 vs 1e-05
377: [ FAILED ] CriterionTest.ViterbiPath (527 ms)
377: [----------] 1 test from CriterionTest (527 ms total)
377:
377: [----------] Global test environment tear-down
377: [==========] 1 test from 1 test suite ran. (527 ms total)
377: [ PASSED ] 0 tests.
377: [ FAILED ] 1 test, listed below:
377: [ FAILED ] CriterionTest.ViterbiPath
377:
377: 1 FAILED TEST
6/23 Test #377: CriterionTest.ViterbiPath .....................................***Failed 0.66 sec
test 378
Start 378: CriterionTest.ASGAlternatingBlanks
378: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.ASGAlternatingBlanks"
378: Test timeout computed to be: 1500
378: I0202 18:57:14.839890 30816 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
378: Note: Google Test filter = CriterionTest.ASGAlternatingBlanks
378: [==========] Running 1 test from 1 test suite.
378: [----------] Global test environment set-up.
378: [----------] 1 test from CriterionTest
378: [ RUN ] CriterionTest.ASGAlternatingBlanks
378: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
378: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 1 vs 1e-05
378: [ FAILED ] CriterionTest.ASGAlternatingBlanks (528 ms)
378: [----------] 1 test from CriterionTest (528 ms total)
378:
378: [----------] Global test environment tear-down
378: [==========] 1 test from 1 test suite ran. (528 ms total)
378: [ PASSED ] 0 tests.
378: [ FAILED ] 1 test, listed below:
378: [ FAILED ] CriterionTest.ASGAlternatingBlanks
378:
378: 1 FAILED TEST
7/23 Test #378: CriterionTest.ASGAlternatingBlanks ............................***Failed 0.66 sec
test 379
Start 379: CriterionTest.ViterbiPathConstrained
379: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.ViterbiPathConstrained"
379: Test timeout computed to be: 1500
379: I0202 18:57:15.504343 63296 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
379: Note: Google Test filter = CriterionTest.ViterbiPathConstrained
379: [==========] Running 1 test from 1 test suite.
379: [----------] Global test environment set-up.
379: [----------] 1 test from CriterionTest
379: [ RUN ] CriterionTest.ViterbiPathConstrained
379: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:361: Failure
379: Expected: (af::max<float>(af::abs(diff))) <= (kEpsilon), actual: 1 vs 1e-05
379: [ FAILED ] CriterionTest.ViterbiPathConstrained (530 ms)
379: [----------] 1 test from CriterionTest (530 ms total)
379:
379: [----------] Global test environment tear-down
379: [==========] 1 test from 1 test suite ran. (530 ms total)
379: [ PASSED ] 0 tests.
379: [ FAILED ] 1 test, listed below:
379: [ FAILED ] CriterionTest.ViterbiPathConstrained
379:
379: 1 FAILED TEST
8/23 Test #379: CriterionTest.ViterbiPathConstrained ..........................***Failed 0.67 sec
test 380
Start 380: CriterionTest.CTCViterbiPathNopaces
380: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.CTCViterbiPathNopaces"
380: Test timeout computed to be: 1500
380: I0202 18:57:16.170237 03648 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
380: Note: Google Test filter = CriterionTest.CTCViterbiPathNopaces
380: [==========] Running 1 test from 1 test suite.
380: [----------] Global test environment set-up.
380: [----------] 1 test from CriterionTest
380: [ RUN ] CriterionTest.CTCViterbiPathNopaces
380: [ OK ] CriterionTest.CTCViterbiPathNopaces (528 ms)
380: [----------] 1 test from CriterionTest (528 ms total)
380:
380: [----------] Global test environment tear-down
380: [==========] 1 test from 1 test suite ran. (528 ms total)
380: [ PASSED ] 1 test.
9/23 Test #380: CriterionTest.CTCViterbiPathNopaces ........................... Passed 0.66 sec
test 381
Start 381: CriterionTest.CTCViterbiPathConstrainedEndWithSpace
381: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.CTCViterbiPathConstrainedEndWithSpace"
381: Test timeout computed to be: 1500
381: I0202 18:57:16.833403 49248 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
381: Note: Google Test filter = CriterionTest.CTCViterbiPathConstrainedEndWithSpace
381: [==========] Running 1 test from 1 test suite.
381: [----------] Global test environment set-up.
381: [----------] 1 test from CriterionTest
381: [ RUN ] CriterionTest.CTCViterbiPathConstrainedEndWithSpace
10/23 Test #381: CriterionTest.CTCViterbiPathConstrainedEndWithSpace ...........***Exception: SegFault 0.75 sec
test 382
Start 382: CriterionTest.CTCViterbiPathConstrainedBeginWithSpace
382: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.CTCViterbiPathConstrainedBeginWithSpace"
382: Test timeout computed to be: 1500
382: I0202 18:57:17.583960 78560 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
382: Note: Google Test filter = CriterionTest.CTCViterbiPathConstrainedBeginWithSpace
382: [==========] Running 1 test from 1 test suite.
382: [----------] Global test environment set-up.
382: [----------] 1 test from CriterionTest
382: [ RUN ] CriterionTest.CTCViterbiPathConstrainedBeginWithSpace
11/23 Test #382: CriterionTest.CTCViterbiPathConstrainedBeginWithSpace .........***Exception: SegFault 0.75 sec
test 383
Start 383: CriterionTest.CTCViterbiPathConstrainedBeginAndEndWithSpace
383: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.CTCViterbiPathConstrainedBeginAndEndWithSpace"
383: Test timeout computed to be: 1500
383: I0202 18:57:18.332630 31072 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
383: Note: Google Test filter = CriterionTest.CTCViterbiPathConstrainedBeginAndEndWithSpace
383: [==========] Running 1 test from 1 test suite.
383: [----------] Global test environment set-up.
383: [----------] 1 test from CriterionTest
383: [ RUN ] CriterionTest.CTCViterbiPathConstrainedBeginAndEndWithSpace
12/23 Test #383: CriterionTest.CTCViterbiPathConstrainedBeginAndEndWithSpace ...***Exception: SegFault 0.75 sec
test 384
Start 384: CriterionTest.FCCCost
384: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.FCCCost"
384: Test timeout computed to be: 1500
384: I0202 18:57:19.082457 96448 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
384: Note: Google Test filter = CriterionTest.FCCCost
384: [==========] Running 1 test from 1 test suite.
384: [----------] Global test environment set-up.
384: [----------] 1 test from CriterionTest
384: [ RUN ] CriterionTest.FCCCost
384: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:516: Failure
384: The difference between loss3Host[0] and 0.0 is 0.26428723335266113, which exceeds kEpsilon, where
384: loss3Host[0] evaluates to 0.26428723335266113,
384: 0.0 evaluates to 0, and
384: kEpsilon evaluates to 9.9999997473787516e-06.
384: [ FAILED ] CriterionTest.FCCCost (525 ms)
384: [----------] 1 test from CriterionTest (525 ms total)
384:
384: [----------] Global test environment tear-down
384: [==========] 1 test from 1 test suite ran. (525 ms total)
384: [ PASSED ] 0 tests.
384: [ FAILED ] 1 test, listed below:
384: [ FAILED ] CriterionTest.FCCCost
384:
384: 1 FAILED TEST
13/23 Test #384: CriterionTest.FCCCost .........................................***Failed 0.66 sec
test 385
Start 385: CriterionTest.FCCJacobian
385: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.FCCJacobian"
385: Test timeout computed to be: 1500
385: I0202 18:57:19.743625 90752 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
385: Note: Google Test filter = CriterionTest.FCCJacobian
385: [==========] Running 1 test from 1 test suite.
385: [----------] Global test environment set-up.
385: [----------] 1 test from CriterionTest
385: [ RUN ] CriterionTest.FCCJacobian
385: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
385: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 1.00002 vs 0.001
385: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
385: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 1 vs 0.001
385: [ FAILED ] CriterionTest.FCCJacobian (533 ms)
385: [----------] 1 test from CriterionTest (533 ms total)
385:
385: [----------] Global test environment tear-down
385: [==========] 1 test from 1 test suite ran. (533 ms total)
385: [ PASSED ] 0 tests.
385: [ FAILED ] 1 test, listed below:
385: [ FAILED ] CriterionTest.FCCJacobian
385:
385: 1 FAILED TEST
14/23 Test #385: CriterionTest.FCCJacobian .....................................***Failed 0.67 sec
test 386
Start 386: CriterionTest.FACCost
386: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.FACCost"
386: Test timeout computed to be: 1500
386: I0202 18:57:20.412477 25760 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
386: Note: Google Test filter = CriterionTest.FACCost
386: [==========] Running 1 test from 1 test suite.
386: [----------] Global test environment set-up.
386: [----------] 1 test from CriterionTest
386: [ RUN ] CriterionTest.FACCost
386: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:555: Failure
386: The difference between loss1Host[0] and log(exp(1.5) + exp(2.5)) is 2.813261687518223, which exceeds kEpsilon, where
386: loss1Host[0] evaluates to 0,
386: log(exp(1.5) + exp(2.5)) evaluates to 2.813261687518223, and
386: kEpsilon evaluates to 9.9999997473787516e-06.
386: [ FAILED ] CriterionTest.FACCost (523 ms)
386: [----------] 1 test from CriterionTest (523 ms total)
386:
386: [----------] Global test environment tear-down
386: [==========] 1 test from 1 test suite ran. (523 ms total)
386: [ PASSED ] 0 tests.
386: [ FAILED ] 1 test, listed below:
386: [ FAILED ] CriterionTest.FACCost
386:
386: 1 FAILED TEST
15/23 Test #386: CriterionTest.FACCost .........................................***Failed 0.66 sec
test 387
Start 387: CriterionTest.FACJacobian
387: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.FACJacobian"
387: Test timeout computed to be: 1500
387: I0202 18:57:21.072078 80992 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
387: Note: Google Test filter = CriterionTest.FACJacobian
387: [==========] Running 1 test from 1 test suite.
387: [----------] Global test environment set-up.
387: [----------] 1 test from CriterionTest
387: [ RUN ] CriterionTest.FACJacobian
387: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
387: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 1.00002 vs 0.001
387: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
387: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 1 vs 0.001
387: [ FAILED ] CriterionTest.FACJacobian (535 ms)
387: [----------] 1 test from CriterionTest (535 ms total)
387:
387: [----------] Global test environment tear-down
387: [==========] 1 test from 1 test suite ran. (535 ms total)
387: [ PASSED ] 0 tests.
387: [ FAILED ] 1 test, listed below:
387: [ FAILED ] CriterionTest.FACJacobian
387:
387: 1 FAILED TEST
16/23 Test #387: CriterionTest.FACJacobian .....................................***Failed 0.67 sec
test 388
Start 388: CriterionTest.ASGCost
388: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.ASGCost"
388: Test timeout computed to be: 1500
388: I0202 18:57:21.743112 23264 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
388: Note: Google Test filter = CriterionTest.ASGCost
388: [==========] Running 1 test from 1 test suite.
388: [----------] Global test environment set-up.
388: [----------] 1 test from CriterionTest
388: [ RUN ] CriterionTest.ASGCost
388: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:612: Failure
388: The difference between loss1Host[0] and -log(0.5) is 0.69314718055994529, which exceeds kEpsilon, where
388: loss1Host[0] evaluates to 0,
388: -log(0.5) evaluates to 0.69314718055994529, and
388: kEpsilon evaluates to 9.9999997473787516e-06.
388: [ FAILED ] CriterionTest.ASGCost (526 ms)
388: [----------] 1 test from CriterionTest (526 ms total)
388:
388: [----------] Global test environment tear-down
388: [==========] 1 test from 1 test suite ran. (526 ms total)
388: [ PASSED ] 0 tests.
388: [ FAILED ] 1 test, listed below:
388: [ FAILED ] CriterionTest.ASGCost
388:
388: 1 FAILED TEST
17/23 Test #388: CriterionTest.ASGCost .........................................***Failed 0.66 sec
test 389
Start 389: CriterionTest.ASGJacobian
389: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.ASGJacobian"
389: Test timeout computed to be: 1500
389: I0202 18:57:22.405203 94816 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
389: Note: Google Test filter = CriterionTest.ASGJacobian
389: [==========] Running 1 test from 1 test suite.
389: [----------] Global test environment set-up.
389: [----------] 1 test from CriterionTest
389: [ RUN ] CriterionTest.ASGJacobian
389: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
389: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 439423 vs 0.001
389: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
389: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 8960.51 vs 0.001
389: [ FAILED ] CriterionTest.ASGJacobian (544 ms)
389: [----------] 1 test from CriterionTest (544 ms total)
389:
389: [----------] Global test environment tear-down
389: [==========] 1 test from 1 test suite ran. (544 ms total)
389: [ PASSED ] 0 tests.
389: [ FAILED ] 1 test, listed below:
389: [ FAILED ] CriterionTest.ASGJacobian
389:
389: 1 FAILED TEST
18/23 Test #389: CriterionTest.ASGJacobian .....................................***Failed 0.68 sec
test 390
Start 390: CriterionTest.LinSegJacobian
390: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.LinSegJacobian"
390: Test timeout computed to be: 1500
390: I0202 18:57:23.085035 42816 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
390: Note: Google Test filter = CriterionTest.LinSegJacobian
390: [==========] Running 1 test from 1 test suite.
390: [----------] Global test environment set-up.
390: [----------] 1 test from CriterionTest
390: [ RUN ] CriterionTest.LinSegJacobian
390: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
390: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 447846 vs 0.001
390: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
390: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 144.892 vs 0.001
390: [ FAILED ] CriterionTest.LinSegJacobian (557 ms)
390: [----------] 1 test from CriterionTest (557 ms total)
390:
390: [----------] Global test environment tear-down
390: [==========] 1 test from 1 test suite ran. (557 ms total)
390: [ PASSED ] 0 tests.
390: [ FAILED ] 1 test, listed below:
390: [ FAILED ] CriterionTest.LinSegJacobian
390:
390: 1 FAILED TEST
19/23 Test #390: CriterionTest.LinSegJacobian ..................................***Failed 0.69 sec
test 391
Start 391: CriterionTest.ASGBatching
391: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.ASGBatching"
391: Test timeout computed to be: 1500
391: I0202 18:57:23.777917 11552 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
391: Note: Google Test filter = CriterionTest.ASGBatching
391: [==========] Running 1 test from 1 test suite.
391: [----------] Global test environment set-up.
391: [----------] 1 test from CriterionTest
391: [ RUN ] CriterionTest.ASGBatching
391: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
391: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 1 vs 1e-06
391: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
391: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 0.499896 vs 1e-06
391: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
391: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 978063 vs 1e-06
391: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
391: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 9.32471e+17 vs 1e-06
391: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
391: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 0.499896 vs 1e-06
391: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
391: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 0.499896 vs 1e-06
391: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
391: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 785085 vs 1e-06
391: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
391: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 0.48717 vs 1e-06
391: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
391: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 2.22048e+08 vs 1e-06
391: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:25: Failure
391: Expected: (af::max<float>(af::abs(val))) <= (precision), actual: 0.499896 vs 1e-06
391: [ FAILED ] CriterionTest.ASGBatching (529 ms)
391: [----------] 1 test from CriterionTest (529 ms total)
391:
391: [----------] Global test environment tear-down
391: [==========] 1 test from 1 test suite ran. (529 ms total)
391: [ PASSED ] 0 tests.
391: [ FAILED ] 1 test, listed below:
391: [ FAILED ] CriterionTest.ASGBatching
391:
391: 1 FAILED TEST
20/23 Test #391: CriterionTest.ASGBatching .....................................***Failed 0.66 sec
test 392
Start 392: CriterionTest.ASGCompareLua
392: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.ASGCompareLua"
392: Test timeout computed to be: 1500
392: I0202 18:57:24.443015 83136 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
392: Note: Google Test filter = CriterionTest.ASGCompareLua
392: [==========] Running 1 test from 1 test suite.
392: [----------] Global test environment set-up.
392: [----------] 1 test from CriterionTest
392: [ RUN ] CriterionTest.ASGCompareLua
392: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:782: Failure
392: The difference between lossHost[i] and expectedLoss[i] is 7.7417464256286621, which exceeds 1e-3, where
392: lossHost[i] evaluates to 0,
392: expectedLoss[i] evaluates to 7.7417464256286621, and
392: 1e-3 evaluates to 0.001.
392: [ FAILED ] CriterionTest.ASGCompareLua (533 ms)
392: [----------] 1 test from CriterionTest (533 ms total)
392:
392: [----------] Global test environment tear-down
392: [==========] 1 test from 1 test suite ran. (533 ms total)
392: [ PASSED ] 0 tests.
392: [ FAILED ] 1 test, listed below:
392: [ FAILED ] CriterionTest.ASGCompareLua
392:
392: 1 FAILED TEST
21/23 Test #392: CriterionTest.ASGCompareLua ...................................***Failed 0.67 sec
test 393
Start 393: CriterionTest.LinSegCompareLua
393: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.LinSegCompareLua"
393: Test timeout computed to be: 1500
393: I0202 18:57:25.111885 62080 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
393: Note: Google Test filter = CriterionTest.LinSegCompareLua
393: [==========] Running 1 test from 1 test suite.
393: [----------] Global test environment set-up.
393: [----------] 1 test from CriterionTest
393: [ RUN ] CriterionTest.LinSegCompareLua
393: /root/flashlight/flashlight/pkg/speech/test/criterion/CriterionTest.cpp:869: Failure
393: The difference between lossHost[i] and expectedLoss[i] is 3.462285041809082, which exceeds 1e-3, where
393: lossHost[i] evaluates to 0,
393: expectedLoss[i] evaluates to 3.462285041809082, and
393: 1e-3 evaluates to 0.001.
393: [ FAILED ] CriterionTest.LinSegCompareLua (526 ms)
393: [----------] 1 test from CriterionTest (526 ms total)
393:
393: [----------] Global test environment tear-down
393: [==========] 1 test from 1 test suite ran. (526 ms total)
393: [ PASSED ] 0 tests.
393: [ FAILED ] 1 test, listed below:
393: [ FAILED ] CriterionTest.LinSegCompareLua
393:
393: 1 FAILED TEST
22/23 Test #393: CriterionTest.LinSegCompareLua ................................***Failed 0.66 sec
test 394
Start 394: CriterionTest.AsgSerialization
394: Test command: /root/flashlight/build/flashlight/pkg/speech/test/CriterionTest "--gtest_filter=CriterionTest.AsgSerialization"
394: Test timeout computed to be: 1500
394: I0202 18:57:25.773786 95456 CachingMemoryManager.cpp:114 CachingMemoryManager recyclingSizeLimit_=18446744073709551615 (16777216.00 TiB) splitSizeLimit_=18446744073709551615 (16777216.00 TiB)
394: Note: Google Test filter = CriterionTest.AsgSerialization
394: [==========] Running 1 test from 1 test suite.
394: [----------] Global test environment set-up.
394: [----------] 1 test from CriterionTest
394: [ RUN ] CriterionTest.AsgSerialization
394: [ OK ] CriterionTest.AsgSerialization (532 ms)
394: [----------] 1 test from CriterionTest (532 ms total)
394:
394: [----------] Global test environment tear-down
394: [==========] 1 test from 1 test suite ran. (532 ms total)
394: [ PASSED ] 1 test.
23/23 Test #394: CriterionTest.AsgSerialization ................................ Passed 0.67 sec
Would love to hear any ideas.
Thanks
Bug Description
I've mentioned this on flashlight repo but haven't heard anything, wondered if I might get a response here?
I have installed flashlight asr and want to do some fine tuning on the RASR model. I have downloaded the Transformer 70M model and prepared my own dataset of an hour of audio. I am following the jupyter notebook "FinetuneCTC.ipynb".
I have built using vcpgk, then via docker, and now just finished building from source, but still having issues running fine tuning. I can run fl_asr_test perfectly fine on the same training set, but fl_asr_tutorial_finetune_ctc crashes as per below exception.
There are 373 items in train.lst and 38 items in valid.lst.
Platform and Hardware
######## Below is the output from ./fl_asr_test
Here's the tail output from make test
I've seen other issues raised similar to this where it was requested that the person who raised the issue run the failed tests individually and return the output, so here it is.