search

k2-fsa / snowfall

Moved to https://github.com/k2-fsa/icefall
Apache License 2.0
143 stars 42 forks source link

Issue in device intersection #229

Closed pzelasko closed 3 years ago

pzelasko commented 3 years ago

I am building the GigaSpeech recipe -- the data part seems is mostly ready, but at the very start of training it fails during intersection -- I'm looking at some things but not sure what's the proper way to debug that, any suggestions?

The recipe is copied from LibriSpeech and practically identical. This is what I'm seeing:

2021-07-08 15:37:46,138 INFO [mmi_att_transformer_train.py:608] No ali_model
2021-07-08 15:37:46,139 INFO [mmi_att_transformer_train.py:640] epoch 0, learning rate 0
2021-07-08 15:37:47,956 INFO [mmi_att_transformer_train.py:274] batch 0, epoch 0/10 global average objf: 1.563327 over 4350.0 frames (100.0% kept), current batch average objf: 1.563326 over 4350 frames (100.0% kept) avg time waiting for batch 0.840s
[F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] [F] /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863:lambda [](int)->void::operator()(int)->void block:[0,0,0], thread: [0,0,0] block:[0,0,0], thread: [1,0,0] block:[0,0,0], thread: [2,0,0] block:[0,0,0], thread: [3,0,0] block:[0,0,0], thread: [4,0,0] block:[0,0,0], thread: [5,0,0] block:[0,0,0], thread: [6,0,0] block:[0,0,0], thread: [7,0,0] block:[0,0,0], thread: [8,0,0] block:[0,0,0], thread: [9,0,0] block:[0,0,0], thread: [10,0,0] block:[0,0,0], thread: [11,0,0] block:[0,0,0], thread: [12,0,0] block:[0,0,0], thread: [13,0,0] block:[0,0,0], thread: [14,0,0] block:[0,0,0], thread: [15,0,0] block:[0,0,0], thread: [16,0,0] block:[0,0,0], thread: [17,0,0] block:[0,0,0], thread: [18,0,0] block:[0,0,0], thread: [19,0,0] block:[0,0,0], thread: [20,0,0] block:[0,0,0], thread: [21,0,0] block:[0,0,0], thread: [22,0,0] block:[0,0,0], thread: [23,0,0] block:[0,0,0], thread: [24,0,0] block:[0,0,0], thread: [25,0,0] block:[0,0,0], thread: [26,0,0] block:[0,0,0], thread: [27,0,0] block:[0,0,0], thread: [28,0,0] block:[0,0,0], thread: [29,0,0] block:[0,0,0], thread: [30,0,0] block:[0,0,0], thread: [31,0,0] block:[0,0,0], thread: [32,0,0] block:[0,0,0], thread: [33,0,0] block:[0,0,0], thread: [34,0,0] block:[0,0,0], thread: [35,0,0] block:[0,0,0], thread: [36,0,0] block:[0,0,0], thread: [37,0,0] block:[0,0,0], thread: [38,0,0] block:[0,0,0], thread: [39,0,0] block:[0,0,0], thread: [40,0,0] block:[0,0,0], thread: [41,0,0] block:[0,0,0], thread: [42,0,0] block:[0,0,0], thread: [43,0,0] block:[0,0,0], thread: [44,0,0] block:[0,0,0], thread: [45,0,0] block:[0,0,0], thread: [46,0,0] block:[0,0,0], thread: [47,0,0] block:[0,0,0], thread: [48,0,0] block:[0,0,0], thread: [49,0,0] block:[0,0,0], thread: [50,0,0] block:[0,0,0], thread: [51,0,0] block:[0,0,0], thread: [52,0,0] block:[0,0,0], thread: [53,0,0] block:[0,0,0], thread: [54,0,0] block:[0,0,0], thread: [55,0,0] block:[0,0,0], thread: [56,0,0] block:[0,0,0], thread: [57,0,0] block:[0,0,0], thread: [58,0,0] block:[0,0,0], thread: [59,0,0] block:[0,0,0], thread: [60,0,0] block:[0,0,0], thread: [61,0,0] block:[0,0,0], thread: [62,0,0] block:[0,0,0], thread: [63,0,0] block:[0,0,0], thread: [64,0,0] block:[0,0,0], thread: [65,0,0] block:[0,0,0], thread: [66,0,0] block:[0,0,0], thread: [67,0,0] block:[0,0,0], thread: [68,0,0] block:[0,0,0], thread: [69,0,0] block:[0,0,0], thread: [70,0,0] block:[0,0,0], thread: [71,0,0] block:[0,0,0], thread: [72,0,0] block:[0,0,0], thread: [73,0,0] block:[0,0,0], thread: [74,0,0] block:[0,0,0], thread: [75,0,0] block:[0,0,0], thread: [76,0,0] block:[0,0,0], thread: [77,0,0] block:[0,0,0], thread: [78,0,0] block:[0,0,0], thread: [79,0,0] block:[0,0,0], thread: [80,0,0] block:[0,0,0], thread: [81,0,0] block:[0,0,0], thread: [82,0,0] block:[0,0,0], thread: [83,0,0] block:[0,0,0], thread: [84,0,0] block:[0,0,0], thread: [85,0,0] block:[0,0,0], thread: [86,0,0] block:[0,0,0], thread: [87,0,0] block:[0,0,0], thread: [88,0,0] block:[0,0,0], thread: [89,0,0] block:[0,0,0], thread: [90,0,0] block:[0,0,0], thread: [91,0,0] block:[0,0,0], thread: [92,0,0] block:[0,0,0], thread: [93,0,0] Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: Check failed: tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_star/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [32,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [33,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [34,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [35,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [36,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [37,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [38,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [39,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [40,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [41,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [42,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [43,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [44,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [45,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [46,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [47,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [48,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [49,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [50,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [51,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [52,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [53,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [54,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [55,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [56,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [57,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [58,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [59,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [60,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [61,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [62,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [63,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [0,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [1,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [2,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [3,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [4,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [5,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [6,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [7,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [8,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [9,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [10,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [11,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [12,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [13,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [14,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [15,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [16,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [17,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [18,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [19,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [20,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [21,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [22,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [23,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [24,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [25,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [26,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [27,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [28,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [29,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [30,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [31,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [64,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [65,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [66,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [67,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [68,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [69,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [70,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [71,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [72,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [73,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [74,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [75,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [76,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [77,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [78,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [79,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [80,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [81,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [82,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [83,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [84,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [85,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [86,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [87,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [88,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [89,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [90,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [91,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [92,0,0] Assertion `Some bad things happened` failed.
/usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/intersect_dense.cu:863: lambda [](int)->void::operator()(int)->void: block: [0,0,0], thread: [93,0,0] Assertion `Some bad things happened` failed.
t) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0tot_score_end == tot_score_start || fabs(tot_score_end - tot_score_start) < 1.0                                                                                              nannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannannan vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs  vs nan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-infnan-inf

[F] /usr/share/miniconda/envs/k2/conda-bld/k2_1625566162510/work/k2/csrc/array.h:341:T k2::Array1<T>::operator[](int32_t) const [with T = int; int32_t = int] Check failed: ret == cudaSuccess (710 vs. 0)  Error: device-side assert triggered. 

[ Stack-Trace: ]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/libk2_log.so(k2::internal::GetStackTrace()+0x4c) [0x2aab3ffce21c]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/libk2context.so(k2::Array1<int>::operator[](int) const+0x6d3) [0x2aab3b362cd3]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/libk2context.so(k2::Renumbering::ComputeOld2New()+0x169) [0x2aab3b35deb9]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/libk2context.so(k2::Renumbering::ComputeNew2Old()+0x9e0) [0x2aab3b35f6c0]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/libk2context.so(k2::MultiGraphDenseIntersect::FormatOutput(k2::Array1<int>*, k2::Array1<int>*)+0x100c) [0x2aab3b4bd0cc]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/libk2context.so(k2::IntersectDense(k2::Ragged<k2::Arc>&, k2::DenseFsaVec&, k2::Array1<int> const*, float, k2::Ragged<k2::Arc>*, k2::Array1<int>*, k2::Array1<int>*)+0x457) [0x2aab3b4ad617]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so(+0x6676f) [0x2aab3a32f76f]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so(+0x1bdaf) [0x2aab3a2e4daf]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(PyCFunction_Call+0x58) [0x5555556a82d8]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyObject_MakeTpCall+0x23c) [0x555555697edc]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalFrameDefault+0x11dd) [0x5555557204ad]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalCodeWithName+0x300) [0x5555556ed760]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyFunction_Vectorcall+0x1e3) [0x5555556ee593]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(PyObject_CallObject+0x52) [0x5555556f3982]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/torch/lib/libtorch_python.so(THPFunction_apply(_object*, _object*)+0x8fd) [0x2aaad2b4e3fd]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(PyCFunction_Call+0xe0) [0x5555556a8360]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyObject_MakeTpCall+0x23c) [0x555555697edc]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalFrameDefault+0x45a9) [0x555555723879]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalCodeWithName+0x300) [0x5555556ed760]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyFunction_Vectorcall+0x1e3) [0x5555556ee593]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x103562) [0x555555657562]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalCodeWithName+0x300) [0x5555556ed760]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyFunction_Vectorcall+0x1e3) [0x5555556ee593]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x103562) [0x555555657562]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyFunction_Vectorcall+0x10b) [0x5555556ee4bb]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x19aac9) [0x5555556eeac9]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(PyObject_Call+0x414) [0x555555698874]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalFrameDefault+0x2088) [0x555555721358]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalCodeWithName+0x300) [0x5555556ed760]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyObject_Call_Prepend+0x181) [0x5555556ef051]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x19b3fa) [0x5555556ef3fa]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyObject_MakeTpCall+0x23c) [0x555555697edc]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalFrameDefault+0x475) [0x55555571f745]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalCodeWithName+0x8b1) [0x5555556edd11]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyFunction_Vectorcall+0x1e3) [0x5555556ee593]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x103562) [0x555555657562]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalCodeWithName+0x300) [0x5555556ed760]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyFunction_Vectorcall+0x1e3) [0x5555556ee593]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x103562) [0x555555657562]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalCodeWithName+0x300) [0x5555556ed760]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyFunction_Vectorcall+0x1e3) [0x5555556ee593]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x103562) [0x555555657562]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyFunction_Vectorcall+0x10b) [0x5555556ee4bb]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x10425f) [0x55555565825f]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(_PyEval_EvalCodeWithName+0x300) [0x5555556ed760]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(PyEval_EvalCode+0x23) [0x5555557824e3]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x22e584) [0x555555782584]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x2547c4) [0x5555557a87c4]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(+0x115620) [0x555555669620]
/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/bin/python(PyRun_SimpleFileExFlags+0x384) [0x55555566c362]

Traceback (most recent call last):
  File "./mmi_att_transformer_train.py", line 738, in <module>
    main()
  File "./mmi_att_transformer_train.py", line 731, in main
    run(rank=0, world_size=1, args=args)
  File "./mmi_att_transformer_train.py", line 641, in run
    objf, valid_objf, global_batch_idx_train = train_one_epoch(
  File "./mmi_att_transformer_train.py", line 251, in train_one_epoch
    curr_batch_objf, curr_batch_frames, curr_batch_all_frames = get_objf(
  File "./mmi_att_transformer_train.py", line 115, in get_objf
    mmi_loss, tot_frames, all_frames = loss_fn(nnet_output, texts, supervision_segments)
  File "/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/exp/pzelasko/snowfall/snowfall/objectives/mmi.py", line 212, in forward
    return func(nnet_output=nnet_output,
  File "/exp/pzelasko/snowfall/snowfall/objectives/mmi.py", line 88, in _compute_mmi_loss_exact_optimized
    num_den_lats = k2.intersect_dense(num_den_reordered_graphs,
  File "/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/k2/autograd.py", line 810, in intersect_dense
    _IntersectDenseFunction.apply(a_fsas, b_fsas, out_fsa, output_beam,
  File "/home/hltcoe/pzelasko/miniconda3/envs/snowfall2/lib/python3.8/site-packages/k2/autograd.py", line 550, in forward
    ragged_arc, arc_map_a, arc_map_b = _k2.intersect_dense(
RuntimeError: Some bad things happed.

@danpovey @csukuangfj

pzelasko commented 3 years ago

FYI the data seems correctly constructed to me, also upon checking random cuts (I added a new storage type in Lhotse so I am also making sure the data pipeline works OK):

image

(in case you're wondering about the dark regions, these cuts are context-less, so they are concatenated just like LibriSpeech, if I can get these working, then I'll start training with the contextual cuts)

(EDIT: actually it's likely I have a bug where a small number of cuts are losing their supervisions; but it's unlikely it triggers the k2 error)

pzelasko commented 3 years ago

OK, nevermind -- it actually turned out that these missing red boxes with supervisions were causing the problem. I'm not exactly sure what went wrong, but when I fixed the data pipeline, the training seems to run fine for ~200 steps now.

danpovey commented 3 years ago

Likely there were NaN's or Inf's in the nnet output.

On Fri, Jul 9, 2021 at 5:58 AM Piotr Żelasko @.***> wrote:

OK, nevermind -- it actually turned out that these missing red boxes with supervisions were causing the problem. I'm not exactly sure what went wrong, but when I fixed the data pipeline, the training seems to run fine for ~200 steps now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/k2-fsa/snowfall/issues/229#issuecomment-876772623, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLOZAMQMNB4XFJSYFLR3TWYNRDANCNFSM5ABMGIMQ .

pzelasko commented 3 years ago

That'd be weird because restoring missing supervisions should not have affected the AM output; I was expecting that the supervision_segments arg to DenseFsaVec may have contained something unexpected, e.g. skipped an example_idx, or had a zero-duration segment, etc. -- but I didn't pursue this since everything worked.

danpovey commented 3 years ago

Hm. Probably it should have failed earlier than that, i.e. we should have detected the problem, whatever it was, earlier.

On Fri, Jul 9, 2021 at 11:36 AM Piotr Żelasko @.***> wrote:

That'd be weird because restoring missing supervisions should not have affected the AM output; I was expecting that the supervision_segments arg to DenseFsaVec may have contained something unexpected, e.g. skipped an example_idx, or had a zero-duration segment, etc. -- but I didn't pursue this since everything worked.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/k2-fsa/snowfall/issues/229#issuecomment-876888037, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO4NOLM7ILBPFZC7QNLTWZVCZANCNFSM5ABMGIMQ .