new error at pruning step

[vince62s]

Dan, I did not update with your last commit but my thought is that it is not related. Here is a log of another error, popping at pruning time (step1, step0 being ok)

float-counts-prune 0.25 200000 data/lm/lm_4_prune4000000.pocolm/work/step0/float.all data/lm/lm_4_prune4000000.pocolm/work/step0/protected.all data/lm/lm_4_prune4000000.pocolm/work/step1/float.1 data/lm/lm_4_prune4000000.pocolm/work/step1/float.2 data/lm/lm_4_prune4000000.pocolm/work/step1/float.3 data/lm/lm_4_prune4000000.pocolm/work/step1/float.4 2>>data/lm/lm_4_prune4000000.pocolm/work/step1/log/float_counts_prune.log float-counts-prune: float-counts-prune.cc:364: float pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float): Assertion `count > 0.0 && discount > 0.0 && backoff_total > backoff_count && backoff_total >= 0.99 * discount' failed.

[danpovey]

I think it could be related. Try the latest commit.

On Wed, Nov 9, 2016 at 2:22 PM, vince62s notifications@github.com wrote:

Dan, I did not update with your last commit but my thought is that it is not related. Here is a log of another error, popping at pruning time (step1, step0 being ok) float-counts-prune 0.25 200000 data/lm/lm_4_prune4000000.pocolm/work/step0/float.all data/lm/lm_4_prune4000000.pocolm/work/step0/protected.all data/lm/lm_4_prune4000000.pocolm/work/step1/float.1 data/lm/lm_4_prune4000000.pocolm/work/step1/float.2 data/lm/lm_4_prune4000000.pocolm/work/step1/float.3 data/lm/lm_4_prune4000000.pocolm/work/step1/float.4 2>>data/lm/lm_4_prune4000000.pocolm/work/step1/log/float_counts_prune.log

float-counts-prune: float-counts-prune.cc:364: float pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float): Assertion `count > 0.0 && discount > 0.0 && backoff_total > backoff_count && backoff_total >= 0.99 * discount' failed.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/danpovey/pocolm/issues/82, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVuxPeUpCkAOOhbyADkhYeBngXg_nrks5q8h1kgaJpZM4Kt5Bg .

[vince62s]

same thing. I also have this in the console, that I did not mention before: warning: in float-counts,1.29348e-11 != 1.29409e-11 warning: in float-counts,9.8279e-12 != 9.83022e-12 warning: in float-counts,8.16296e-12 != 8.16658e-12 warning: in float-counts,9.1713e-13 != 9.22712e-13 warning: in float-counts,7.59186e-13 != 7.63044e-13 warning: in float-counts,8.89627e-13 != 8.89721e-13

[danpovey]

Can you run the binary that failed in gdb?

On Wed, Nov 9, 2016 at 3:45 PM, vince62s notifications@github.com wrote:

same thing. I also have this in the console, that I did not mention before: warning: in float-counts,1.29348e-11 != 1.29409e-11 warning: in float-counts,9.8279e-12 != 9.83022e-12 warning: in float-counts,8.16296e-12 != 8.16658e-12 warning: in float-counts,9.1713e-13 != 9.22712e-13 warning: in float-counts,7.59186e-13 != 7.63044e-13 warning: in float-counts,8.89627e-13 != 8.89721e-13

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danpovey/pocolm/issues/82#issuecomment-259521734, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu3jg_rZdOukwNl6QXJ3aO4_UQkAeks5q8jD4gaJpZM4Kt5Bg .

[vince62s]

I need some guidance, never done that before. float-counts-prune is called in the pruning python script.

[danpovey]

Do

gdb --args float-counts-prune 0.25 200000
data/lm/lm_4_prune4000000.pocolm/work/step0/float.all
data/lm/lm_4_prune4000000.pocolm/work/step0/protected.all
data/lm/lm_4_prune4000000.pocolm/work/step1/float.1
data/lm/lm_4_prune4000000.pocolm/work/step1/float.2
data/lm/lm_4_prune4000000.pocolm/work/step1/float.3
data/lm/lm_4_prune4000000.pocolm/work/step1/float.4

(gdb) r
.. wait till it crashes
(gdb) up
keep doing 'up' till you get to the right stack frame, or 'down' if you go
too far
then
(gdb)  p count
(gdb) p discount
.. and any other relevant variables.

On Wed, Nov 9, 2016 at 4:11 PM, vince62s notifications@github.com wrote:

I need some guidance, never done that before. float-counts-prune is called in the pruning python script.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danpovey/pocolm/issues/82#issuecomment-259527791, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu-hcR9whpcKLbiXWc-QWDbvaL9nSks5q8jcSgaJpZM4Kt5Bg .

[vince62s]

(gdb) up

4 0x00000000004038d0 in pocolm::FloatCountsPruner::PruningLogprobChange (

this=0x7fffffffd890, count=0.500001132, discount=2.55000576e-10, 
backoff_count=0.500001132, backoff_total=0.500001132)
at float-counts-prune.cc:362

362 assert(count > 0.0 && discount > 0.0 && (gdb) p count $1 = 0.500001132 (gdb) p discount $2 = 2.55000576e-10 (gdb)

[danpovey]

Can you type 'bt' ? I want to see the whole backtrace.

On Wed, Nov 9, 2016 at 4:38 PM, vince62s notifications@github.com wrote:

(gdb) up

4 https://github.com/danpovey/pocolm/pull/4 0x00000000004038d0 in

pocolm::FloatCountsPruner::PruningLogprobChange ( this=0x7fffffffd890, count=0.500001132, discount=2.55000576e-10, backoff_count=0.500001132, backoff_total=0.500001132) at float-counts-prune.cc:362 362 assert(count > 0.0 && discount > 0.0 && (gdb) p count $1 = 0.500001132 (gdb) p discount $2 = 2.55000576e-10 (gdb)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danpovey/pocolm/issues/82#issuecomment-259533855, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu3BjNFNg37PknvxZwH0TPcij16iUks5q8j1NgaJpZM4Kt5Bg .

[vince62s]

(gdb) bt

0 0x00007ffff7221c37 in **GI_raise (sig=sig@entry=6)

at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

1 0x00007ffff7225028 in __GI_abort () at abort.c:89

2 0x00007ffff721abf6 in __assert_fail_base (

fmt=0x7ffff736b3b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
assertion=assertion@entry=0x418220 "count > 0.0 && discount > 0.0 && backoff_total > backoff_count && backoff_total >= 0.99 \* discount", 
file=file@entry=0x417d20 "float-counts-prune.cc", line=line@entry=364, 
function=function@entry=0x418a80 <pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)::__PRETTY_FUNCTION**> "float pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)") at assert.c:92

3 0x00007ffff721aca2 in **GI___assert_fail (

assertion=0x418220 "count > 0.0 && discount > 0.0 && backoff_total > backoff_count && backoff_total >= 0.99 \* discount", 
file=0x417d20 "float-counts-prune.cc", line=364, 
function=0x418a80 <pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)::__PRETTY_FUNCTION**> "float pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)") at assert.c:101

4 0x00000000004038d0 in pocolm::FloatCountsPruner::PruningLogprobChange (

this=0x7fffffffd890, count=0.500001132, discount=2.55000576e-10, 
backoff_count=0.500001132, backoff_total=0.500001132)
at float-counts-prune.cc:362

5 0x0000000000404094 in pocolm::FloatCountsPruner::DoPruningForLmState (

---Type to continue, or q to quit--- this=0x7fffffffd890, history_length=3) at float-counts-prune.cc:537

6 0x0000000000403be6 in pocolm::FloatCountsPruner::FlushOutput (

this=0x7fffffffd890, history_length=3) at float-counts-prune.cc:476

7 0x0000000000403532 in pocolm::FloatCountsPruner::ProcessInput (

this=0x7fffffffd890) at float-counts-prune.cc:279

8 0x0000000000402939 in pocolm::FloatCountsPruner::FloatCountsPruner (

this=0x7fffffffd890, argc=9, argv=0x7fffffffde58)
at float-counts-prune.cc:178

9 0x0000000000401761 in main (argc=9, argv=0x7fffffffde58)

at float-counts-prune.cc:701

(gdb)

[danpovey]

I'm a little confused how the discount amount would be so tiny although the count for the state was reasonable (0.5). Can you show the metaparameters (final.metaparams)? The only way the metaparameter optimization would have reached this result is if your dev set was part of your training set.

On Wed, Nov 9, 2016 at 4:50 PM, vince62s notifications@github.com wrote:

(gdb) bt

0 0x00007ffff7221c37 in

_GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

1 https://github.com/danpovey/pocolm/issues/1 0x00007ffff7225028 in

GI_abort () at abort.c:89 #2 https://github.com/danpovey/pocolm/issues/2 0x00007ffff721abf6 in __assert_fail_base ( fmt=0x7ffff736b3b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x418220 "count > 0.0 && discount > 0.0 && backoff_total > backoff_count && backoff_total >= 0.99 * discount", file=file@entry=0x417d20 "float-counts-prune.cc", line=line@entry=364, function=function@entry=0x418a80 <pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)::PRETTYFUNCTION> "float pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)") at assert.c:92

3 https://github.com/danpovey/pocolm/issues/3 0x00007ffff721aca2 in

_GI_assert_fail ( assertion=0x418220 "count > 0.0 && discount > 0.0 && backoff_total > backoff_count && backoff_total >= 0.99 * discount", file=0x417d20 "float-counts-prune.cc", line=364, function=0x418a80 <pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)::PRETTYFUNCTION> "float pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)") at assert.c:101

4 https://github.com/danpovey/pocolm/pull/4 0x00000000004038d0 in

pocolm::FloatCountsPruner::PruningLogprobChange ( this=0x7fffffffd890, count=0.500001132, discount=2.55000576e-10, backoff_count=0.500001132, backoff_total=0.500001132) at float-counts-prune.cc:362

5 https://github.com/danpovey/pocolm/issues/5 0x0000000000404094 in

pocolm::FloatCountsPruner::DoPruningForLmState ( ---Type to continue, or q to quit--- this=0x7fffffffd890, history_length=3) at float-counts-prune.cc:537

6 https://github.com/danpovey/pocolm/issues/6 0x0000000000403be6 in

pocolm::FloatCountsPruner::FlushOutput ( this=0x7fffffffd890, history_length=3) at float-counts-prune.cc:476

7 https://github.com/danpovey/pocolm/pull/7 0x0000000000403532 in

pocolm::FloatCountsPruner::ProcessInput ( this=0x7fffffffd890) at float-counts-prune.cc:279

8 https://github.com/danpovey/pocolm/issues/8 0x0000000000402939 in

pocolm::FloatCountsPruner::FloatCountsPruner ( this=0x7fffffffd890, argc=9, argv=0x7fffffffde58) at float-counts-prune.cc:178

9 https://github.com/danpovey/pocolm/pull/9 0x0000000000401761 in main

(argc=9, argv=0x7fffffffde58) at float-counts-prune.cc:701 (gdb)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danpovey/pocolm/issues/82#issuecomment-259536589, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVuymVWayXX7Qf5Ab7kQMz3FGjP33nks5q8kAvgaJpZM4Kt5Bg .

[vince62s]

yes, the dev is in the training set

count_scale_1 0.500001125302356 order2_D1 0.724208510369568 order2_D2 0.297727220253901 order2_D3 0.127992215348989 order2_D4 0.055617652988870 order3_D1 0.055193203110626 order3_D2 0.013962291974136 order3_D3 0.005097823583461 order3_D4 0.001982142055876 order4_D1 0.000000000100000 order4_D2 0.000000000010000 order4_D3 0.000000000001000 order4_D4 0.000000000000100

[vince62s]

would that behave differently if I use the fold into train option ?

[danpovey]

You will get totally wrong discounting parameters if the dev is in the training set. Yes, you should exclude it and do fold-into-dev. Jian, could you modify optimize_metaparameters.py so that if the highest-order D1 after optimization finishes is less than, say, 0.1, it exits with error status after printing out a message saying "your dev set is probably in your training set; this is not advisable"?

On Wed, Nov 9, 2016 at 4:59 PM, vince62s notifications@github.com wrote:

would that behave differently if I use the fold into train option ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danpovey/pocolm/issues/82#issuecomment-259538683, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu5e5KIdLNm8QYcOfNioR1s1JaZs0ks5q8kI3gaJpZM4Kt5Bg .

[wantee]

sure.

On Thu, Nov 10, 2016 at 6:12 AM, Daniel Povey notifications@github.com wrote:

You will get totally wrong discounting parameters if the dev is in the training set. Yes, you should exclude it and do fold-into-dev. Jian, could you modify optimize_metaparameters.py so that if the highest-order D1 after optimization finishes is less than, say, 0.1, it exits with error status after printing out a message saying "your dev set is probably in your training set; this is not advisable"?

On Wed, Nov 9, 2016 at 4:59 PM, vince62s notifications@github.com wrote:

would that behave differently if I use the fold into train option ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danpovey/pocolm/issues/82#issuecomment-259538683, or mute the thread https://github.com/notifications/unsubscribe-auth/ ADJVu5e5KIdLNm8QYcOfNioR1s1JaZs0ks5q8kI3gaJpZM4Kt5Bg

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/danpovey/pocolm/issues/82#issuecomment-259541663, or mute the thread https://github.com/notifications/unsubscribe-auth/AGphTPx6vdU83j8eMMHqv2swT7I1nUeUks5q8kVGgaJpZM4Kt5Bg .

[francisr]

A related question: what should I do if I don't have a dev set?

[wantee]

A dev set is required for optimizing the metaparameters. You could extract part of your training set as dev set.