ttystudio not terminating

[schenker]

on Linux Mint 17

~$ ttystudio test.gif --log
...
writing frame 268 - 7 left
writing frame 269 - 6 left
writing frame 270 - 5 left
writing frame 271 - 4 left
writing frame 272 - 3 left
writing frame 273 - 2 left
writing frame 274 - 1 left
writing eof

it stopped there. After ~20 Minutes i canceled it. The resulting gif file is incomplete.

[chjj]

Hmmm, can you try:

$ ttystudio frames.json
^Q
$ ttystudio frames.json output.gif --log

If it outputs a gif successfully this might help point out the issue.

[schenker]

same problem

[chjj]

Hmm, what node version? It's possible the stream isn't flushing/finishing properly, or ttystudio isn't listening for it properly. Also, do you see high cpu usage when it hangs, or does it just idle?

[schenker]

the node version is v0.10.25. CPU Usage is high.

[srid99]

I also have the same problem

$ps aux | grep ttystudio
sri      17293 99.9  6.6 1190236 538640 pts/10 Rl+  19:57  95:42 node /home/sri/.npm-packages/bin/ttystudio output.gif --log

Just wanted to try the tool so went with the basic step and recorded for around 10 seconds and it took more than 2 hours and still it is running. I tried it on Ubuntu 14.04 with Node v0.10.37

[chjj]

It should almost instantly finish after writing the eof byte. It sounds like this might be a node v0.10.x problem.

The fact that there is high cpu confused me even more. I expected it to be idling.

Could one of you guys give me a quick strace of recording a 1 second gif?

[schenker]

creating a gif with 8 frames works for me now, but it took 95 Seconds on an Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz. strace here https://gist.github.com/schenker/cb713fe66b6a5de34cb7

[chjj]

it took 95 Seconds on an Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz.

Yikes. That sounds like some kind of stream flushing issue. Once the EOF is written, ttystudio's work is done. The rest is up to node to flush the stream. I'll see if I can reproduce this on v0.10.x.

[lafolle]

Same issue, my node version v0.10.29

[chjj]

Cannot reproduce with v0.10.33.

[jottr]

I am seeing this issue with

node --version                                                                                                                                                                           
v0.11.13

on OSX in zsh.

[xamox]

Same issue as well.

Ubuntu 15.04 3.19.0-22-generic node - v0.12.6 zsh - 5.0.7 npm install of ttystudio

[rm3nchaca]

same issue on fedora 20 node v0.10.36

the flushing to the gif file is very slow ~1k per minute

[chjj]

on OSX in zsh

Another theory is toppled. I thought this had something to do with ubuntu/debian-based distros for a second. Cannot reproduce on OSX either. Can you brew upgrade nodejs and see what happens with v0.12.x?

[chjj]

Just a thought to everyone: does the process also hang when writing a .png?

[chjj]

Any chance I could get a gist of a full strace log, or at least starting from the write to fd 2 of "writing eof"? All of those mmaps and munmaps in the above strace are interesting. I don't see those in my strace log. Perhaps a large garbage collection pause by v8?

[srid99]

Tried png format and got error

vagrant@vagrant-ubuntu-trusty-64:~$ ttystudio output.png --log
initializing writer
writing image
building palette
writing head
TypeError: offset is not uint
    at TypeError (<anonymous>)
    at checkInt (buffer.js:707:11)
    at Buffer.writeUInt32BE (buffer.js:748:5)
    at ExpandingBuffer.(anonymous function) [as writeUInt32BE] (/home/vagrant/.npm-packages/lib/node_modules/ttystudio/lib/buffer.js:27:34)
    at PNGWriter.write32 (/home/vagrant/.npm-packages/lib/node_modules/ttystudio/lib/png.js:133:14)
    at PNGWriter.writeHead (/home/vagrant/.npm-packages/lib/node_modules/ttystudio/lib/png.js:225:8)
    at PNGWriter.write (/home/vagrant/.npm-packages/lib/node_modules/ttystudio/lib/png.js:60:8)
    at SGRWriter.write (/home/vagrant/.npm-packages/lib/node_modules/ttystudio/lib/writer.js:53:22)
    at compile (/home/vagrant/.npm-packages/lib/node_modules/ttystudio/bin/ttystudio:61:10)
    at /home/vagrant/.npm-packages/lib/node_modules/ttystudio/bin/ttystudio:93:12

The output.frames.json is created (that was pretty fast) and then it takes some time (again one of the core is utilized 100%) then fails with above error message.

[chjj]

@srid99, that is very interesting. I see why the png writer is failing. It looks like different node versions handle buffer error checks very differently.

@srid99, please try the latest version and tell me what happens with a png. That error should be fixed since things are a little more strict about buffer writing now. Sorry about that.

[srid99]

@chjj I tried with Node 0.12.x and I don't see the issue anymore (with gif output format). The processing is fast now. But I get a different issue. I only get a partial output (I recorded for 10 seconds but the output is for 3 seconds).

Between with the png format I don't see the animation happening.

[dklesev]

same issue. my params:

• node --version
  • v0.12.7
• fish --version
  • fish, version 2.1.0

GIF mode works but it hangs on building palette

it seems to work but it takes a loooooooot of time for PNG on i7 (maybe adding multi-threading would solve this hmm?!)

[chjj]

@srid @srid99, well, at least there's some improvement there. This definitely sounds like some kind of stream flushing issue. ttystudio is killing the process before the write stream can finish writing to the file. I really wish I could reproduce this and do some debugging, but we need to figure out the exact conditions of what's causing this. It doesn't seem to discriminate by OS or node.js version. I'm at a loss.

If anyone who can reproduce this wants to dig into the code and figure out what's causing this or how to fix it, it would be much appreciated. I suggest starting here: https://github.com/chjj/ttystudio/blob/v0.0.6/lib/gif.js#L250

@dklesev, there is no "building palette" message for gifs. Maybe you're referring to PNGs? The reason it takes so long is because PNGs are optimized to use a global palette (whereas gifs use a local color table for each frame), so it has to go in and look at every color in every frame to build the palette (this can be optimized more so it ignores the useless frames that don't really update the screen). If you don't care about size optimization, the --rgba option is likely faster.

PNG frames themselves take longer to write than GIFs likely due to the DEFLATE compression. LZW for gifs seems to be a bit faster (oddly enough, the LZW implementation is in javascript and DEFLATE is just calling a linked zlib function).

[dklesev]

@chjj yes I meant APNG and yes, --rgba is much faster

[chjj]

@srid @srid99, would you mind trying the latest commit? I think I solved the issue with prematurely writing out incomplete files. Thanks.

[srid]

@chjj ask @srid99 and not me :-)

[srid]

Oh, now I see who snapped the srid.in domain before me!

[chjj]

Whoops, sorry @srid. :)

cc @srid99

[chjj]

After some messing around, I'm pretty sure ttystudio will no longer cut off gifs/apngs. Let me know if this problem still exists with the latest commit.

[srid99]

@chjj Thanks for the quick response and fixes. I tested it with the last commit and I still have the issue.

And noticed the gif is not working in Chrome but works fine with Firefox. So indeed ttystudio renders the complete gif and it is Chrome which doesn't load the complete gif. Not sure if it is Chrome related problem or we have to enhance our way of gif output.

I am using Chrome Version 43.0.2357.132 (64-bit) and Ubuntu 14.04 Any of you guys have it working with Chrome ?

[chjj]

@srid99, I figured out what it was: some browsers (or maybe my implementation) seem to have trouble with color table lengths of: 2, 4, and 64. Lengths of 8, 16, 32, 128, and 256 seem fine though. That was what was cutting off the gif (you maybe only saw 1/3 of the total animation I'm guessing? I managed to reproduce that eventually).

As far as the writing eof hanging, I seemed to have fixed that by not prematurely exiting the process while node frantically tries to flush everything. Then again, with all the buffered data in the stream, it may just take a while. I'm not sure on this one, but I think it should all flush naturally now.

I'll publish my new fix for this in a minute. I'm fairly certain it should work. Let me know if you have any more trouble. Marking closed for now.

As an aside: I've also implemented an attempt to optimize gifs more by creating a global color palette for gifs, which reduces the size of my gif by at least 20kb.