search

Me-TV / Me-TV

It's TV for me computer.
GNU General Public License v3.0
52 stars 10 forks source link

me-tv-record – control recording quality to allow control of resource usage #34

Open dougK44883 opened 5 years ago

dougK44883 commented 5 years ago

Nonsense explanation, limited trials. I tested me-tv-record to record a 30min show. I noticed it didn't stop!? I tried ^C... just kept repeating ^C - had no effect. Used system-monitor to kill-process. file wouldn't play :( Tried same again with --duration 02 success! Screwy test --duration 10 yielded non-stop recording with non-playable file? This is all I've got, doesn't make sense <10 is limited >10 is infinite?

It may be my older system, eating up ~98% CPU (ok) but 1.5gb memory usage is exorbitant! Comparably, me-tv peaks ~12% CPU to display 1080i channel.

russel commented 5 years ago

Problem noted. I have to admit I didn't do any >10 minute testing, just 2, 3 an d5 minutes. I fear I may not be able to get stuck into this one properly till 2019-01-01 once back at home: tomorrow and Monday are travel days so no chance of getting the portable office out of it's packing.

I'll definitely have to check the resource usage, it should definitely not be that high at all.

dougK44883 commented 5 years ago

I just can't let it go... I have some channels HD either 720p or 1080i, while some of the sub-channels are SD, (standard definition) you know, original TV. What is now sometimes called 480i probably for consistency. Ok, so more OCD trial and error research and development I checked recorded different resolution.

GST_DEBUG=3 me-tv-record --channel 11.2 --duration 10 --output TenMinuteShow.mpeg

The television defines it as 480i ~30 -19% CPU while memory was ~435mMiB and stopped with error

0:10:00.804210755  4854 0x7f92fe9bd780 ERROR                  libav :0:: ac-tex damaged at 28 20
0:10:00.804300028  4854 0x7f92fe8b2f70 ERROR                  libav :0:: Warning MVs not available

Playable video player shows Length: 00:09:53 (it takes 7 seconds for the video to actually start?) OK, up arrow and changed to channel 11.1 HD 1080i (sometimes playing 1080i vs 720p has artifacts) I didn't as accurately record these CPU was ~98% and memory ~1.5GiB for me-tv-record ?! It did finish successfully.... Length: 00:09:54

Now a 720p channel, ~95-88%CPU ~605MiB memory. Stopping again with error

0:10:11.442489492  6189 0x7fca8ab6bb90 ERROR                  libav :0:: ac-tex damaged at 44 30
0:10:11.442535622  6189 0x7fca8aac3f70 ERROR                  libav :0:: Warning MVs not available

Playable video Length: 00:09:53 This would suggest video resolution, encoding(?) makes CPU usage quite variable. And memory usage as well. I believe me-tv-record started somewhat as a proof-of-concept for you, so obviously there's no if [ -e filename ] are you sure type checking. FYI, I can enclose the file name in "quotes" and add, for example $(date '+ - %a %b %d') I suppose $(date '+ - %a %d %b') is actually proper :) ^C functions as well

Don't know what did/not happen earlier today? I'd like hope I didn't over-react. The very first time I did use --verbose, not saying that kept it going. I made sure the time via properties dialog from system-monitor displaying start time, nearly 30min for a 20min duration ...the age old - just one of those things? (yea, I'm not buying it either but want to move forward) I have (unusual) restarted my computer this afternoon (that sounds windows-ish). Sorry for any inconvenience. Still, I haven't left it record a full 30min program of any resolution. But, again/still, it's trending positively.

date.abd.TXT

dougK44883 commented 5 years ago

This is a screwy and seemingly random issue. Again, tried me-tv-record. Added time to the file name so I wouldn't get distracted when I started --output "show$(date '+ - %H_%M').mpeg" gives Hour_Minute. After 15 I struggled with ^C 

^C
^C
^C^C^X^C^C^C^X^C^C
0:15:11.771468244 13048 0x7fca36949240 ERROR                  libav :0:: overread 12
0:15:11.771613615 13048 0x7fca368b2f70 ERROR                  libav :0:: Warning MVs not available

Maybe I was too impatient, and one of those did work. Since the mpeg is playable, it probably took time to close down. The time from GST_DBEUG shows 15 minutes - while the mpeg is Length: 00:09:53. So, unfortunately... now it's more that just one... or those things. I'm looking for methods to monitor/log memory and CPU usage for various resolutions.

russel commented 5 years ago

I just tried recording 15 mins of BBC NEWS and got a 15 min MP4 file. However, if I fast forward the file from the beginning to about 10mins things get very stressed and my workstation seizes up with both Glade and Totem, but both are GStreamer based so the problem may be there. Further investigation tomorrow.

Update: It seems to play fine, all 14:59 of it. The problem is there was some signal drop out in the middle and fast forward over it couldn't cope. So it seems I can't reproduce the problem of longer records that you have seen. :-(

russel commented 5 years ago

Sorry for the delay in getting to this one.

  1. Ctrl-C should work to terminate the execution if it is running in the foreground of a shell, and should leave a playable file. I'll have more of a play, but 3.0.7 has worked for me.

  2. The code does appear to be using far more resources than it should/was expected. There seem to be far more kernel threads than I anticipated. I shall have to seek advice on this from the GStreamer folk.

russel commented 5 years ago

It seems that each GStreamer processing element in a pipeline tries to use as many threads as there are processors, which a priori is not a silly thing to do. Apparently this can be amended to mitigate the CPU issues. However there is a primary culprit: the x264enc encoder tries, by default, to use the highest possible quality and so uses huge amounts of CPU. This is likely the important place to do mitigation. I will see if I can add a video quality factor to the me-tv-record and me-tv-schedule commands to try things out.

dougK44883 commented 5 years ago

Ctrl-C should work to terminate the execution if it is running in the foreground of a shell, and should leave a playable file. I'll have more of a play, but 3.0.7 has worked for me.

I suspect some of my issue was due to the high CPU usage or response time. I may have been impatient and expected a quicker action. Unlamented, with repeated key-presses making me 'wait' it did end with a playable file.

russel commented 5 years ago

Ctrl-C does take a while on me-tv-record I'm afraid – it is terminating the data stream for correct encoding rather than severe prejudice! I suspect the high quality encoding of x264enc is integral to that problem.

dougK44883 commented 5 years ago

I suspect the high quality encoding of x264enc is integral to that problem.

I had hesitated asking/suggesting if it were possible to reduce encoding resolution. But then, I couldn't say it was a problem or possible.

This past week I trialed some more. Sorry I can't recall specifics (and just deleted recordings). Recordings from channels at 1080i (resolution 1920x1080) takes all the CPU and 1.5Gb ram. Couldn't recall specifics, 720 channels (1280x720) and 480 (720x480) was overall moderate usage. Noticeably lower CPU and RAM usages was ~500 -~650.

It may be a preference or depend on your system capabilities, of course I like higher quality, but it's not a deal-breaker. Really, some of these shows aren't important enough for finite detail.

russel commented 5 years ago

I think the issue is how to present "desired quality" easily on the me-tv-record and me-tv-schedule command lines. The x264enc has a very complicated API because the encoding algorithm appears to be very complicated – assuming I am interpreting the documentation correctly. Presenting this complexity on the command line is just the wrong thing to do. However, just at the moment, I do not understand h.264 well enough to make a judgement call.

The resolution issue you are mentioning is certainly an aspect of quality and hence computation cost, but there are multiple (sometimes overlapping, sometimes not) issues of the encoding process that are also factors. Ones I do not properly understand just at the moment. Research required.

russel commented 5 years ago

@dougK44883 Should we close this issue as having too many bits to it, and open new ones to be more focused, or do we change the title on this one to focus on controlling the level of quality so as to control the level of resource use?

dougK44883 commented 5 years ago

do we change the title on this one to focus on controlling the level of quality so as to control the level of resource use?

Yes... to either. I did some recordings last week, didn't keep document as intended so I discarded but I believe the "drug-out" time may be related to resources. If/when it did eventually stop, the actual recording was a few seconds shy of it's duration, despite how long it actually took.

So yes, the primary issue or greatest concern from my perspective is controlling quality level.