problems about SIM-M training

[RichardLee777]

I have some problems with the results of training SIM-M. The position acc of time and theatre_name are always 0. I set it according to your README.md. May I still need to rewrite something, or change a parameter, please?

output/pred_res.test.00000000.json date: joint slot acc: 0.244868, class acc: 0.339443, position acc: 0.0227273 movie: joint slot acc: 0.0857771, class acc: 0.186217, position acc: 0 time: joint slot acc: 0.11217, class acc: 0.136364, position acc: 0 num_tickets: joint slot acc: 0.0175953, class acc: 0.729472, position acc: 0.0241206 theatre_name: joint slot acc: 0.00219941, class acc: 0.349707, position acc: 0 output/pred_res.test.00001000.json date: joint slot acc: 0.942815, class acc: 0.999267, position acc: 0.918691 movie: joint slot acc: 0.416422, class acc: 0.876833, position acc: 0.390291 time: joint slot acc: 0.328446, class acc: 0.985337, position acc: 0 num_tickets: joint slot acc: 0.994135, class acc: 0.998534, position acc: 0.993958 theatre_name: joint slot acc: 0.184751, class acc: 0.994868, position acc: 0 output/pred_res.test.00002000.json date: joint slot acc: 0.942082, class acc: 1, position acc: 0.916579 movie: joint slot acc: 0.426686, class acc: 0.936217, position acc: 0.374437 time: joint slot acc: 0.335044, class acc: 0.990469, position acc: 0 num_tickets: joint slot acc: 0.993402, class acc: 0.996334, position acc: 0.99596 theatre_name: joint slot acc: 0.187683, class acc: 0.997801, position acc: 0 output/pred_res.test.00003000.json date: joint slot acc: 0.943548, class acc: 1, position acc: 0.918691 movie: joint slot acc: 0.432551, class acc: 0.923754, position acc: 0.387569 time: joint slot acc: 0.335044, class acc: 0.991202, position acc: 0 num_tickets: joint slot acc: 0.995601, class acc: 0.998534, position acc: 0.995972 theatre_name: joint slot acc: 0.187683, class acc: 0.997801, position acc: 0 output/pred_res.test.00004000.json date: joint slot acc: 0.943548, class acc: 1, position acc: 0.918691 movie: joint slot acc: 0.428886, class acc: 0.906891, position acc: 0.391223 time: joint slot acc: 0.334311, class acc: 0.990469, position acc: 0 num_tickets: joint slot acc: 0.993402, class acc: 0.996334, position acc: 0.99596 theatre_name: joint slot acc: 0.187683, class acc: 0.997801, position acc: 0 output/pred_res.test.00005000.json date: joint slot acc: 0.931818, class acc: 1, position acc: 0.901795 movie: joint slot acc: 0.429619, class acc: 0.907625, position acc: 0.391791 time: joint slot acc: 0.332845, class acc: 0.989003, position acc: 0 num_tickets: joint slot acc: 0.993402, class acc: 0.996334, position acc: 0.99596 theatre_name: joint slot acc: 0.187683, class acc: 0.997801, position acc: 0 output/pred_res.test.00006000.json date: joint slot acc: 0.942082, class acc: 1, position acc: 0.916579 movie: joint slot acc: 0.42522, class acc: 0.917889, position acc: 0.381215 time: joint slot acc: 0.334311, class acc: 0.990469, position acc: 0 num_tickets: joint slot acc: 0.993402, class acc: 0.996334, position acc: 0.99596 theatre_name: joint slot acc: 0.187683, class acc: 0.997801, position acc: 0 output/pred_res.test.00007000.json date: joint slot acc: 0.944282, class acc: 1, position acc: 0.919747 movie: joint slot acc: 0.433284, class acc: 0.929619, position acc: 0.385662 time: joint slot acc: 0.335044, class acc: 0.991202, position acc: 0 num_tickets: joint slot acc: 0.993402, class acc: 0.998534, position acc: 0.992951 theatre_name: joint slot acc: 0.188416, class acc: 0.998534, position acc: 0 output/pred_res.test.00008000.json date: joint slot acc: 0.943548, class acc: 1, position acc: 0.918691 movie: joint slot acc: 0.433284, class acc: 0.935484, position acc: 0.382883 time: joint slot acc: 0.335044, class acc: 0.991202, position acc: 0 num_tickets: joint slot acc: 0.997067, class acc: 1, position acc: 0.99598 theatre_name: joint slot acc: 0.187683, class acc: 0.997801, position acc: 0 output/pred_res.test.00009000.json date: joint slot acc: 0.943548, class acc: 1, position acc: 0.918691 movie: joint slot acc: 0.432551, class acc: 0.928886, position acc: 0.385104 time: joint slot acc: 0.335044, class acc: 0.991202, position acc: 0 num_tickets: joint slot acc: 0.993402, class acc: 0.998534, position acc: 0.992951 theatre_name: joint slot acc: 0.187683, class acc: 0.997801, position acc: 0 output/pred_res.test.00010000.json date: joint slot acc: 0.943548, class acc: 0.999267, position acc: 0.919747 movie: joint slot acc: 0.433284, class acc: 0.918622, position acc: 0.390984 time: joint slot acc: 0.335044, class acc: 0.991202, position acc: 0 num_tickets: joint slot acc: 0.995601, class acc: 0.998534, position acc: 0.995972 theatre_name: joint slot acc: 0.187683, class acc: 0.997801, position acc: 0 output/pred_res.test.00011000.json date: joint slot acc: 0.943548, class acc: 0.999267, position acc: 0.919747 movie: joint slot acc: 0.432551, class acc: 0.918622, position acc: 0.390064 time: joint slot acc: 0.335044, class acc: 0.991202, position acc: 0 num_tickets: joint slot acc: 0.995601, class acc: 0.998534, position acc: 0.995972 theatre_name: joint slot acc: 0.187683, class acc: 0.997801, position acc: 0 output/pred_res.test.00012000.json date: joint slot acc: 0.943548, class acc: 1, position acc: 0.918691 movie: joint slot acc: 0.432551, class acc: 0.918622, position acc: 0.390064 time: joint slot acc: 0.335044, class acc: 0.991202, position acc: 0 num_tickets: joint slot acc: 0.997067, class acc: 1, position acc: 0.99598 theatre_name: joint slot acc: 0.187683, class acc: 0.997801, position acc: 0 Joint goal acc: 0.0909091, output/pred_res.test.00002000.json Joint goal acc: 0.0909091, output/pred_res.test.00003000.json Joint goal acc: 0.0909091, output/pred_res.test.00007000.json Joint goal acc: 0.0909091, output/pred_res.test.00008000.json Joint goal acc: 0.0909091, output/pred_res.test.00009000.json Joint goal acc: 0.0909091, output/pred_res.test.00010000.json Joint goal acc: 0.0909091, output/pred_res.test.00011000.json Joint goal acc: 0.0909091, output/pred_res.test.00012000.json Joint goal acc: 0.090176, output/pred_res.test.00004000.json Joint goal acc: 0.090176, output/pred_res.test.00005000.json Joint goal acc: 0.090176, output/pred_res.test.00006000.json Joint goal acc: 0.0894428, output/pred_res.test.00001000.json Joint goal acc: 0, output/pred_res.test.00000000.json

[lukecq1231]

@guanlinchao I have the same problem. Could you give us some suggestions?

[guanlinchao]

How about increase the value for FLAGS.class_loss_ratio? Is "token_loss" for these two slots decreasing?

[RichardLee777]

@guanlinchao Thank you for your reply. But position acc of time and theatre_name are always 0. Could it be a problem with data reading?

[RichardLee777]

The result of SIM-R has the same problem. The position acc of time is always 0.

output/pred_res.test.00001000.json category: joint slot acc: 0.826834, class acc: 0.830035, position acc: 0.904348 rating: joint slot acc: 0.930733, class acc: 0.930733, position acc: nan num_people: joint slot acc: 0.983993, class acc: 0.991851, position acc: 0.980797 location: joint slot acc: 0.707218, class acc: 0.859721, position acc: 0.304117 restaurant_name: joint slot acc: 0.485739, class acc: 0.923166, position acc: 0.0265544 time: joint slot acc: 0.532014, class acc: 0.956927, position acc: 0 date: joint slot acc: 0.901921, class acc: 1, position acc: 0.774431 price_range: joint slot acc: 0.798603, class acc: 0.842258, position acc: 0.622166 meal: joint slot acc: 0.91298, class acc: 0.91298, position acc: nan output/pred_res.test.00002000.json category: joint slot acc: 0.970023, class acc: 0.980792, position acc: 0.941824 rating: joint slot acc: 0.930733, class acc: 0.930733, position acc: nan num_people: joint slot acc: 0.996799, class acc: 0.998545, position acc: 0.995733 location: joint slot acc: 0.768626, class acc: 1, position acc: 0.352606 restaurant_name: joint slot acc: 0.491269, class acc: 0.925786, position acc: 0.0254569 time: joint slot acc: 0.532014, class acc: 0.957218, position acc: 0 date: joint slot acc: 0.902212, class acc: 1, position acc: 0.7751 price_range: joint slot acc: 0.912398, class acc: 0.984866, position acc: 0.704276 meal: joint slot acc: 0.915017, class acc: 0.922584, position acc: 0.212121 output/pred_res.test.00003000.json category: joint slot acc: 0.98312, class acc: 0.98603, position acc: 0.984615 rating: joint slot acc: 0.930733, class acc: 0.94383, position acc: 0.150943 num_people: joint slot acc: 0.996508, class acc: 0.998836, position acc: 0.99431 location: joint slot acc: 0.768044, class acc: 0.997963, position acc: 0.352989 restaurant_name: joint slot acc: 0.475553, class acc: 0.92986, position acc: 0.0194724 time: joint slot acc: 0.571013, class acc: 0.996217, position acc: 0 date: joint slot acc: 0.901048, class acc: 0.998545, position acc: 0.775017 price_range: joint slot acc: 0.914726, class acc: 0.986903, position acc: 0.706509 meal: joint slot acc: 0.938882, class acc: 0.940338, position acc: 0.934211 output/pred_res.test.00004000.json category: joint slot acc: 0.982829, class acc: 0.98603, position acc: 0.983077 rating: joint slot acc: 0.944994, class acc: 0.974098, position acc: 0.346405 num_people: joint slot acc: 0.995634, class acc: 0.999709, position acc: 0.990043 location: joint slot acc: 0.768917, class acc: 1, position acc: 0.35342 restaurant_name: joint slot acc: 0.486903, class acc: 0.91851, position acc: 0.0165782 time: joint slot acc: 0.571013, class acc: 0.996217, position acc: 0 date: joint slot acc: 0.901048, class acc: 0.99709, position acc: 0.777628 price_range: joint slot acc: 0.9156, class acc: 0.986903, position acc: 0.710059 meal: joint slot acc: 0.962456, class acc: 0.978754, position acc: 0.718593 output/pred_res.test.00005000.json category: joint slot acc: 0.98312, class acc: 0.98603, position acc: 0.984615 rating: joint slot acc: 0.944121, class acc: 0.980501, position acc: 0.277457 num_people: joint slot acc: 0.996799, class acc: 0.999709, position acc: 0.992888 location: joint slot acc: 0.768917, class acc: 0.999418, position acc: 0.353997 restaurant_name: joint slot acc: 0.485448, class acc: 0.926077, position acc: 0.0156047 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.90163, class acc: 0.999418, position acc: 0.774799 price_range: joint slot acc: 0.917346, class acc: 0.989232, position acc: 0.707692 meal: joint slot acc: 0.976135, class acc: 0.991269, position acc: 0.738693 output/pred_res.test.00006000.json category: joint slot acc: 0.983702, class acc: 0.986612, position acc: 0.984615 rating: joint slot acc: 0.953725, class acc: 0.989523, position acc: 0.272189 num_people: joint slot acc: 0.99709, class acc: 0.998836, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.488359, class acc: 0.914144, position acc: 0.0168011 time: joint slot acc: 0.571886, class acc: 0.99709, position acc: 0 date: joint slot acc: 0.90163, class acc: 0.999418, position acc: 0.774799 price_range: joint slot acc: 0.914726, class acc: 0.986903, position acc: 0.706509 meal: joint slot acc: 0.973807, class acc: 0.988941, position acc: 0.738693 output/pred_res.test.00007000.json category: joint slot acc: 0.983702, class acc: 0.986612, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.297753 num_people: joint slot acc: 0.997672, class acc: 0.999709, position acc: 0.995021 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.488068, class acc: 0.908033, position acc: 0.0163599 time: joint slot acc: 0.571886, class acc: 0.99709, position acc: 0 date: joint slot acc: 0.902212, class acc: 1, position acc: 0.7751 price_range: joint slot acc: 0.922002, class acc: 0.994179, position acc: 0.706509 meal: joint slot acc: 0.975553, class acc: 0.990687, position acc: 0.736041 output/pred_res.test.00008000.json category: joint slot acc: 0.988068, class acc: 0.990978, position acc: 0.984615 rating: joint slot acc: 0.960128, class acc: 0.996508, position acc: 0.277457 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.770373, class acc: 1, position acc: 0.357492 restaurant_name: joint slot acc: 0.48865, class acc: 0.912689, position acc: 0.0155405 time: joint slot acc: 0.571013, class acc: 0.996217, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.927532, class acc: 1, position acc: 0.705325 meal: joint slot acc: 0.976135, class acc: 0.990687, position acc: 0.74359 output/pred_res.test.00009000.json category: joint slot acc: 0.988068, class acc: 0.990978, position acc: 0.984615 rating: joint slot acc: 0.959255, class acc: 0.995634, position acc: 0.297753 num_people: joint slot acc: 0.99709, class acc: 1, position acc: 0.992888 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.492724, class acc: 0.92142, position acc: 0.0245033 time: joint slot acc: 0.571013, class acc: 0.995925, position acc: 0 date: joint slot acc: 0.902212, class acc: 1, position acc: 0.7751 price_range: joint slot acc: 0.926077, class acc: 0.997963, position acc: 0.707692 meal: joint slot acc: 0.974389, class acc: 0.988941, position acc: 0.74359 output/pred_res.test.00010000.json category: joint slot acc: 0.989523, class acc: 0.992433, position acc: 0.984615 rating: joint slot acc: 0.958964, class acc: 0.995343, position acc: 0.297753 num_people: joint slot acc: 0.996508, class acc: 0.999709, position acc: 0.992176 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.495052, class acc: 0.920838, position acc: 0.031767 time: joint slot acc: 0.570431, class acc: 0.995634, position acc: 0 date: joint slot acc: 0.902212, class acc: 1, position acc: 0.7751 price_range: joint slot acc: 0.927823, class acc: 0.999709, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991269, position acc: 0.746193 output/pred_res.test.00011000.json category: joint slot acc: 0.993306, class acc: 0.996217, position acc: 0.984615 rating: joint slot acc: 0.958091, class acc: 0.993888, position acc: 0.293103 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.490978, class acc: 0.917055, position acc: 0.0207358 time: joint slot acc: 0.571013, class acc: 0.996217, position acc: 0 date: joint slot acc: 0.901921, class acc: 1, position acc: 0.774431 price_range: joint slot acc: 0.926659, class acc: 0.998545, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991851, position acc: 0.738693 output/pred_res.test.00012000.json category: joint slot acc: 0.99447, class acc: 0.997381, position acc: 0.984615 rating: joint slot acc: 0.958091, class acc: 0.99447, position acc: 0.297753 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.49156, class acc: 0.91851, position acc: 0.022 time: joint slot acc: 0.571595, class acc: 0.996799, position acc: 0 date: joint slot acc: 0.901339, class acc: 1, position acc: 0.773092 price_range: joint slot acc: 0.926368, class acc: 0.998254, position acc: 0.707692 meal: joint slot acc: 0.976135, class acc: 0.990687, position acc: 0.746193 output/pred_res.test.00013000.json category: joint slot acc: 0.993888, class acc: 0.996799, position acc: 0.984615 rating: joint slot acc: 0.958964, class acc: 0.995343, position acc: 0.297753 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.489814, class acc: 0.916182, position acc: 0.0180965 time: joint slot acc: 0.571595, class acc: 0.996799, position acc: 0 date: joint slot acc: 0.901921, class acc: 1, position acc: 0.774431 price_range: joint slot acc: 0.926659, class acc: 0.998545, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991851, position acc: 0.738693 output/pred_res.test.00014000.json category: joint slot acc: 0.992724, class acc: 0.995634, position acc: 0.984615 rating: joint slot acc: 0.958673, class acc: 0.995052, position acc: 0.297753 num_people: joint slot acc: 0.99709, class acc: 0.999709, position acc: 0.993599 location: joint slot acc: 0.770373, class acc: 1, position acc: 0.357492 restaurant_name: joint slot acc: 0.494761, class acc: 0.926077, position acc: 0.0288336 time: joint slot acc: 0.571304, class acc: 0.996508, position acc: 0 date: joint slot acc: 0.902212, class acc: 1, position acc: 0.7751 price_range: joint slot acc: 0.928114, class acc: 0.999709, position acc: 0.708876 meal: joint slot acc: 0.973225, class acc: 0.986612, position acc: 0.747253 output/pred_res.test.00015000.json category: joint slot acc: 0.993015, class acc: 0.996217, position acc: 0.983077 rating: joint slot acc: 0.959255, class acc: 0.995634, position acc: 0.297753 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.491269, class acc: 0.920838, position acc: 0.0250991 time: joint slot acc: 0.570722, class acc: 0.995925, position acc: 0 date: joint slot acc: 0.901921, class acc: 1, position acc: 0.774431 price_range: joint slot acc: 0.92695, class acc: 0.998545, position acc: 0.708876 meal: joint slot acc: 0.976426, class acc: 0.99156, position acc: 0.738693 output/pred_res.test.00016000.json category: joint slot acc: 0.991269, class acc: 0.99447, position acc: 0.983077 rating: joint slot acc: 0.959255, class acc: 0.993888, position acc: 0.30814 num_people: joint slot acc: 0.997672, class acc: 0.999418, position acc: 0.995733 location: joint slot acc: 0.770373, class acc: 1, position acc: 0.357492 restaurant_name: joint slot acc: 0.486903, class acc: 0.901921, position acc: 0.011781 time: joint slot acc: 0.571304, class acc: 0.996508, position acc: 0 date: joint slot acc: 0.901921, class acc: 1, position acc: 0.774431 price_range: joint slot acc: 0.928114, class acc: 0.999709, position acc: 0.708876 meal: joint slot acc: 0.976135, class acc: 0.990687, position acc: 0.746193 output/pred_res.test.00017000.json category: joint slot acc: 0.992142, class acc: 0.995052, position acc: 0.984615 rating: joint slot acc: 0.959255, class acc: 0.995634, position acc: 0.297753 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.770373, class acc: 1, position acc: 0.357492 restaurant_name: joint slot acc: 0.490105, class acc: 0.906286, position acc: 0.0192044 time: joint slot acc: 0.571013, class acc: 0.996217, position acc: 0 date: joint slot acc: 0.902212, class acc: 1, position acc: 0.7751 price_range: joint slot acc: 0.927823, class acc: 0.999709, position acc: 0.707692 meal: joint slot acc: 0.975844, class acc: 0.990396, position acc: 0.74359 output/pred_res.test.00018000.json category: joint slot acc: 0.993306, class acc: 0.996217, position acc: 0.984615 rating: joint slot acc: 0.959546, class acc: 0.995925, position acc: 0.297753 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.770373, class acc: 1, position acc: 0.357492 restaurant_name: joint slot acc: 0.493015, class acc: 0.927241, position acc: 0.0292778 time: joint slot acc: 0.571304, class acc: 0.996508, position acc: 0 date: joint slot acc: 0.902212, class acc: 1, position acc: 0.7751 price_range: joint slot acc: 0.928114, class acc: 0.999709, position acc: 0.708876 meal: joint slot acc: 0.976426, class acc: 0.990978, position acc: 0.746193 output/pred_res.test.00019000.json category: joint slot acc: 0.99156, class acc: 0.99447, position acc: 0.984615 rating: joint slot acc: 0.959255, class acc: 0.995634, position acc: 0.297753 num_people: joint slot acc: 0.997381, class acc: 0.999127, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.493597, class acc: 0.928987, position acc: 0.0310881 time: joint slot acc: 0.571886, class acc: 0.99709, position acc: 0 date: joint slot acc: 0.902212, class acc: 1, position acc: 0.7751 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.975262, class acc: 0.989814, position acc: 0.74359 output/pred_res.test.00020000.json category: joint slot acc: 0.99156, class acc: 0.99447, position acc: 0.984615 rating: joint slot acc: 0.958964, class acc: 0.995343, position acc: 0.297753 num_people: joint slot acc: 0.997672, class acc: 0.999709, position acc: 0.995021 location: joint slot acc: 0.769499, class acc: 0.999709, position acc: 0.355863 restaurant_name: joint slot acc: 0.485739, class acc: 0.907159, position acc: 0.0116041 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928696, class acc: 1, position acc: 0.710059 meal: joint slot acc: 0.975262, class acc: 0.990978, position acc: 0.725888 output/pred_res.test.00021000.json category: joint slot acc: 0.992724, class acc: 0.995634, position acc: 0.984615 rating: joint slot acc: 0.959255, class acc: 0.995634, position acc: 0.277457 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.769499, class acc: 0.999709, position acc: 0.355863 restaurant_name: joint slot acc: 0.486612, class acc: 0.905995, position acc: 0.00894085 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928696, class acc: 1, position acc: 0.710059 meal: joint slot acc: 0.974098, class acc: 0.989814, position acc: 0.723077 output/pred_res.test.00022000.json category: joint slot acc: 0.992724, class acc: 0.995634, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.769208, class acc: 0.999709, position acc: 0.355049 restaurant_name: joint slot acc: 0.48603, class acc: 0.908324, position acc: 0.0109066 time: joint slot acc: 0.571304, class acc: 0.996508, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928696, class acc: 1, position acc: 0.710059 meal: joint slot acc: 0.97468, class acc: 0.990396, position acc: 0.725888 output/pred_res.test.00023000.json category: joint slot acc: 0.992142, class acc: 0.995052, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.490396, class acc: 0.917055, position acc: 0.0220147 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902212, class acc: 1, position acc: 0.7751 price_range: joint slot acc: 0.928696, class acc: 1, position acc: 0.710059 meal: joint slot acc: 0.975844, class acc: 0.990396, position acc: 0.746193 output/pred_res.test.00024000.json category: joint slot acc: 0.992724, class acc: 0.995634, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.997381, class acc: 0.999127, position acc: 0.995733 location: joint slot acc: 0.769499, class acc: 0.999709, position acc: 0.355863 restaurant_name: joint slot acc: 0.486321, class acc: 0.909488, position acc: 0.0128988 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 0.999709, position acc: 0.708876 meal: joint slot acc: 0.975553, class acc: 0.990105, position acc: 0.74359 output/pred_res.test.00025000.json category: joint slot acc: 0.992142, class acc: 0.995052, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.997381, class acc: 0.999127, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.498836, class acc: 0.958091, position acc: 0.0442156 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928696, class acc: 1, position acc: 0.710059 meal: joint slot acc: 0.976717, class acc: 0.991269, position acc: 0.746193 output/pred_res.test.00026000.json category: joint slot acc: 0.992433, class acc: 0.995634, position acc: 0.983077 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.486321, class acc: 0.910943, position acc: 0.0128552 time: joint slot acc: 0.572468, class acc: 0.997672, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928696, class acc: 1, position acc: 0.710059 meal: joint slot acc: 0.976717, class acc: 0.991269, position acc: 0.746193 output/pred_res.test.00027000.json category: joint slot acc: 0.992433, class acc: 0.995634, position acc: 0.983077 rating: joint slot acc: 0.957509, class acc: 0.993888, position acc: 0.281609 num_people: joint slot acc: 0.997672, class acc: 0.999418, position acc: 0.995733 location: joint slot acc: 0.770373, class acc: 1, position acc: 0.357492 restaurant_name: joint slot acc: 0.486321, class acc: 0.911816, position acc: 0.0128292 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928405, class acc: 1, position acc: 0.708876 meal: joint slot acc: 0.976135, class acc: 0.990687, position acc: 0.74359 output/pred_res.test.00028000.json category: joint slot acc: 0.990687, class acc: 0.993597, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.997963, class acc: 0.999709, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.496799, class acc: 0.934517, position acc: 0.0340398 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.975553, class acc: 0.991269, position acc: 0.725888 output/pred_res.test.00029000.json category: joint slot acc: 0.991851, class acc: 0.994761, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.998254, class acc: 1, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.497672, class acc: 0.933353, position acc: 0.0360592 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991851, position acc: 0.738693 output/pred_res.test.00030000.json category: joint slot acc: 0.992433, class acc: 0.995343, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.998254, class acc: 1, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.497672, class acc: 0.933353, position acc: 0.0360592 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991851, position acc: 0.738693 output/pred_res.test.00031000.json category: joint slot acc: 0.993306, class acc: 0.996217, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.998254, class acc: 1, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.496799, class acc: 0.930151, position acc: 0.0343709 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991851, position acc: 0.738693 output/pred_res.test.00032000.json category: joint slot acc: 0.991851, class acc: 0.994761, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.998254, class acc: 1, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.496508, class acc: 0.92986, position acc: 0.0337443 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991269, position acc: 0.746193 output/pred_res.test.00033000.json category: joint slot acc: 0.993306, class acc: 0.996217, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.998254, class acc: 1, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.495634, class acc: 0.927241, position acc: 0.0319843 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991269, position acc: 0.746193 output/pred_res.test.00034000.json category: joint slot acc: 0.992724, class acc: 0.995634, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.998254, class acc: 1, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.494761, class acc: 0.925786, position acc: 0.0301244 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991851, position acc: 0.738693 output/pred_res.test.00035000.json category: joint slot acc: 0.993306, class acc: 0.996217, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.998254, class acc: 1, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.495343, class acc: 0.926659, position acc: 0.0313725 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991851, position acc: 0.738693 output/pred_res.test.00036000.json category: joint slot acc: 0.993306, class acc: 0.996217, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.998254, class acc: 1, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.495343, class acc: 0.92695, position acc: 0.0313521 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991851, position acc: 0.738693 output/pred_res.test.00037000.json category: joint slot acc: 0.993306, class acc: 0.996217, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.998254, class acc: 1, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.495925, class acc: 0.927532, position acc: 0.0326158 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991851, position acc: 0.738693 output/pred_res.test.00038000.json category: joint slot acc: 0.993306, class acc: 0.996217, position acc: 0.984615 rating: joint slot acc: 0.958382, class acc: 0.994761, position acc: 0.269006 num_people: joint slot acc: 0.998254, class acc: 1, position acc: 0.995733 location: joint slot acc: 0.76979, class acc: 1, position acc: 0.355863 restaurant_name: joint slot acc: 0.495925, class acc: 0.927532, position acc: 0.0326158 time: joint slot acc: 0.572177, class acc: 0.997381, position acc: 0 date: joint slot acc: 0.902503, class acc: 1, position acc: 0.77577 price_range: joint slot acc: 0.928114, class acc: 1, position acc: 0.707692 meal: joint slot acc: 0.976717, class acc: 0.991851, position acc: 0.738693 Joint goal acc: 0.198778, output/pred_res.test.00029000.json Joint goal acc: 0.198778, output/pred_res.test.00030000.json Joint goal acc: 0.198778, output/pred_res.test.00031000.json Joint goal acc: 0.198778, output/pred_res.test.00032000.json Joint goal acc: 0.198778, output/pred_res.test.00033000.json Joint goal acc: 0.198778, output/pred_res.test.00037000.json Joint goal acc: 0.198778, output/pred_res.test.00038000.json Joint goal acc: 0.198487, output/pred_res.test.00025000.json Joint goal acc: 0.198487, output/pred_res.test.00035000.json Joint goal acc: 0.198487, output/pred_res.test.00036000.json Joint goal acc: 0.198196, output/pred_res.test.00034000.json Joint goal acc: 0.197905, output/pred_res.test.00010000.json Joint goal acc: 0.197905, output/pred_res.test.00028000.json Joint goal acc: 0.197614, output/pred_res.test.00018000.json Joint goal acc: 0.19674, output/pred_res.test.00019000.json Joint goal acc: 0.196449, output/pred_res.test.00023000.json Joint goal acc: 0.196158, output/pred_res.test.00014000.json Joint goal acc: 0.196158, output/pred_res.test.00026000.json Joint goal acc: 0.195576, output/pred_res.test.00008000.json Joint goal acc: 0.195576, output/pred_res.test.00017000.json Joint goal acc: 0.195576, output/pred_res.test.00024000.json Joint goal acc: 0.195285, output/pred_res.test.00020000.json Joint goal acc: 0.195285, output/pred_res.test.00027000.json Joint goal acc: 0.194994, output/pred_res.test.00012000.json Joint goal acc: 0.194994, output/pred_res.test.00015000.json Joint goal acc: 0.194994, output/pred_res.test.00021000.json Joint goal acc: 0.194994, output/pred_res.test.00022000.json Joint goal acc: 0.194703, output/pred_res.test.00009000.json Joint goal acc: 0.194703, output/pred_res.test.00011000.json Joint goal acc: 0.194703, output/pred_res.test.00013000.json Joint goal acc: 0.194703, output/pred_res.test.00016000.json Joint goal acc: 0.19092, output/pred_res.test.00007000.json Joint goal acc: 0.185972, output/pred_res.test.00006000.json Joint goal acc: 0.184226, output/pred_res.test.00005000.json Joint goal acc: 0.183062, output/pred_res.test.00004000.json Joint goal acc: 0.171711, output/pred_res.test.00003000.json Joint goal acc: 0.158033, output/pred_res.test.00002000.json Joint goal acc: 0.10943, output/pred_res.test.00001000.json

[RichardLee777]

Is the sim-m and sim-r I downloaded different from the version you used? https://github.com/google-research-datasets/simulated-dialogue