Standard output not being captured

[EricR86]

Original report (BitBucket issue) by Rachel Chan (Bitbucket: rcwchan).

---

I noticed that standard output is no longer being captured (eg the traindir/output/o folder). I tested this using simpleseg on changeset 37e5b7a0acd003386c0c8a1fbd2917777bd57d4d (prior to the GMM and validation pull requests). Standard error is fine.

In particular, the files exist but are empty:

#!bash

rachelc [ /scratch/segway/test/simpleseg/test-20170626.LVZIYG/traindir/output/o/0 ]$ ll
total 0
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.0.0.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.0.bundle.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.1.0.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.1.bundle.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.2.0.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.2.bundle.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.3.0.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.3.bundle.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.4.0.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.4.bundle.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:16 emt0.5.0.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:17 emt0.5.bundle.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:17 emt0.6.0.traindir.cfb56fd85a8a11e78abf989096b2d04e
-rw-r--r-- 1 rachelc mhoffma1 0 Jun 26 12:17 emt0.6.bundle.traindir.cfb56fd85a8a11e78abf989096b2d04e

the touchstones seems to expect content there:

#!bash

rachelc [ /scratch/segway/test/simpleseg/touchstone/traindir/output/o/0 ]$ ll
total 56
-rw-r--r-- 1 rachelc mhoffma1 172 Jun 26 12:15 emt0.0.0.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 117 Jun 26 12:15 emt0.0.bundle.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 170 Jun 26 12:15 emt0.1.0.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 117 Jun 26 12:15 emt0.1.bundle.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 170 Jun 26 12:15 emt0.2.0.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 117 Jun 26 12:15 emt0.2.bundle.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 170 Jun 26 12:15 emt0.3.0.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 117 Jun 26 12:15 emt0.3.bundle.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 170 Jun 26 12:15 emt0.4.0.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 117 Jun 26 12:15 emt0.4.bundle.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 170 Jun 26 12:15 emt0.5.0.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 117 Jun 26 12:15 emt0.5.bundle.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 170 Jun 26 12:15 emt0.6.0.traindir.(%[0-9a-f]{32}%)
-rw-r--r-- 1 rachelc mhoffma1 117 Jun 26 12:15 emt0.6.bundle.traindir.(%[0-9a-f]{32}%)

but the tests pass anyway (which may be a separate issue--it's possible that comparisons to empty return TRUE for some reason).

An example of an expected output is:

#!bash

rachelc [ /scratch/segway/test/simpleseg/touchstone/traindir/output/o/0 ]$ cat "emt0.0.0.traindir.(%[0-9a-f]{32}%)"
Segment 0, after Island, log(prob(evidence)) = -8715.890764, per frame =-1.089486, per numUFrams = -1.089486
____ PROGRAM ENDED SUCCESSFULLY WITH STATUS 0 AT (%[^_]+%)____

[EricR86]

Original comment by Eric Roberts (Bitbucket: ericr86, GitHub: ericr86).

---

I've confirmed this behavior and I've traced the first bad commit to 7ce987d70edd72d9773ea9a5c2d921678f15149e. To double check I also confirmed that a3b9724778e8b5e8a2a23855bd113aa7284869bf (the commit before), works as intended. It's not clear from the changeset what causes the change.

[EricR86]

Original comment by Eric Roberts (Bitbucket: ericr86, GitHub: ericr86).

---

• changed priority from "minor" to "major"

[EricR86]

Original comment by Eric Roberts (Bitbucket: ericr86, GitHub: ericr86).

---

I suspect since it was just changing training to be submitted as a task, and that stderr is captured correctly, that the same mechanism used for stderr should be used for stdout and should have been updated with this change but wasn't.

[EricR86]

Original comment by Eric Roberts (Bitbucket: ericr86, GitHub: ericr86).

---

• changed state from "new" to "resolved"

Resolved after changeset bc1b1093b5d1712e67fc251220165d93af3e2b8f