SchedulesDirect 3 weeks of epg data

[weston1968]

Schedules direct has a longer number of days for epg data. Is is possible the change the current 14 day maximum for recording calculation to the Schedules direct maximum number of days. Sometimes I get ahead of myself and add a favorite 18 days out that has already been added.

[enternoescape]

@Narflex If I'm not mistaken, this can be done by just changing the value of LOOKAHEAD in Carny to 23*24*60*60*1000L. With the optimizations I will be committing shortly, increasing this range should have very little performance impact to Carny. My only follow-up concern would be the amount of time this might add to Scheduler.

Edit: 23 days is the most days I think I've seen. We could just adjust for it and then modify the UI so people can go up to 23 days if they want.

[Narflex]

While there is a constant in Carny for it...that's just the default value, this can be modified in Sage.properties via "scheduling_lookahead" which is in milliseconds. So people can try it out first...see if it's OK and then go from there. I think there's even a setting in the UI in Detailed Setup->Advanced to manually adjust it too....OK, well it is in the UI, but that menu limits it to a 14 day max. :) So try it out in the properties file and see how that goes...but I don't foresee any problems, so we can change that to 21 later for everybody.

On Mon, May 15, 2017 at 4:58 PM, Joseph Shuttlesworth < notifications@github.com> wrote:

@Narflex https://github.com/narflex If I'm not mistaken, this can be done by just changing the value of LOOKAHEAD in Carny to 232460601000L. With the optimizations I will be committing shortly, increasing this range should have very little performance impact to Carny. My only follow-up concern would be the amount of time this might add to Scheduler.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/287#issuecomment-301637172, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDBY3LoFn95PVnlDcx2lML24ctWJWks5r6OavgaJpZM4NbwvK .

-- Jeffrey Kardatzke jkardatzke@google.com Google, Inc.

[enternoescape]

Oh yeah, now I see what you're talking about. I didn't look at how that value is actually applied.

[enternoescape]

I know that might sound a little odd, but I didn't make any changes to that specific piece of code.

[enternoescape]

In the new implementation without CPU throttling, it goes from just under 25 seconds to just under 29 seconds on my test setup using the wiz.bin you provided me.

[JustFred51]

I'd like to request that the default be left at it's current value of 3 days. I've got a couple of systems that always seemed to get upset whenever the lookahead value was increased beyond that, and wouldn't like to see them suddenly break. No objections to increasing the 14-day limit.

[weston1968]

Hi Joseph,

Worked like a charm. Just multiplied by 1.5.

thanks Lance

On Mon, May 15, 2017 at 9:55 PM, Keith Fischer notifications@github.com wrote:

I'd like to request that the default be left at it's current value of 3 days. I've got a couple of systems that always seemed to get upset whenever the lookahead value was increased beyond that, and wouldn't like to see them suddenly break. No objections to increasing the 14-day limit.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/287#issuecomment-301653266, or mute the thread https://github.com/notifications/unsubscribe-auth/AbW4gCNX6d-A9dibaTwTdi41uKX_WZBpks5r6QIegaJpZM4NbwvK .

-- "The only thing necessary for the triumph of evil, is for good men to do nothing." - Edmund Burke 1729 – 1797

[enternoescape]

@JustFred51 I'm curious how things will perform for you when we have a new release with the new Carny optimizations. I used to have some issues with all of my favorites disappearing from the schedule in v7 and had the same thing happen in v9. I wish I knew as much as a I know now about how everything ties together because I feel like I could have narrowed the problem down and fixed it. I haven't had any issues in over a year now, so I don't know if I'll ever get the opportunity again.

[JustFred51]

Looking forward to the new Carny. I've got nearly 8 years of recording history in wiz.bin (dating back to Sage6 days). Will admit that there's a bit of apprehension with the increased use of Java heap. As it is, one of my systems (w/3- HD200's) constantly teeters on the edge of instability... I have to restart the server every night to keep the heap in check. OT: Looking forward to the reduction in memory usage that's been made with regard to string de-duplication. Thinking even bigger: wish I understood what would be required to move Sage (Win) to a 64-bit Java environment and be rid of the heap-size limitations. That's a project I'd be willing to help with.

[weston1968]

Hi Keith,

Every few months I run out of memory. I have 4 HD300's. I have been using them about five years. I guess I should do a weekly reboot.

Lance

On Tue, May 16, 2017 at 12:00 AM, Keith Fischer notifications@github.com wrote:

Looking forward to the new Carny. I've got nearly 8 years of recording history in wiz.bin (dating back to Sage6 days). Will admit that there's a bit of apprehension with the increased use of Java heap. As it is, one of my systems (w/3- HD200's) constantly teeters on the edge of instability... I have to restart the server every night to keep the heap in check. OT: Looking forward to the reduction in memory usage that's been made with regard to string de-duplication. Thinking even bigger: wish I understood what would be required to move Sage (Win) to a 64-bit Java environment and be rid of the heap-size limitations. That's a project I'd be willing to help with.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/287#issuecomment-301669161, or mute the thread https://github.com/notifications/unsubscribe-auth/AbW4gKnFEItFZUCJ1uH2_CfZEMWF5e5xks5r6R9rgaJpZM4NbwvK .

-- "The only thing necessary for the triumph of evil, is for good men to do nothing." - Edmund Burke 1729 – 1797

[enternoescape]

The additional memory usage is only while Carny is running, but just about anything you do with any recording will cause it to run again, so the memory might as well be considered always in use. Also you can turn the maps off and still get multi-threaded performance, that's close to linear per core dedicated to the task. I have been looking at more efficient memory allocation strategies for the Show objects.

For my own curiosity, I packed the Show object into individual efficient off heap ByteBuffers and with the maps the total heap size was 354MB (vs 420MB). This of course means that any time you want to do anything with a show, the part that you want needs to be converted back into an object which does create a GC situation, but the reality is that you're not looking at 100% of the shows in the database at any time, so why do we always have all of the data always loaded? When I tried this, I was careful to maintain the identity of the objects since SageTV relies heavily on it, just all of the internals were always being re-created when retrieved. In the UI, I was unable to perceive any significant delays due to the way the data was being loaded.

For what it's worth, I do a lot of my testing on a Windows 7 box. The standard SageTV setup is actually fairly light on memory per client connection. It's definitly the graphic rich UI's causing many of the problems we see. Also I left the Comskip plugin running once while I was doing profiling and it is constantly opening and closing it's own configuration file which creates a not insignificant amount of GC on it's own.

[enternoescape]

That modification does make Carny go from sub 30 seconds to 2 minutes mind you. And before you say something like you're just putting the memory load somewhere else, using on heap ByteBuffers resulted in 393MB of total heap usage, so the structures were more efficient.

[weston1968]

Hi Joseph,

I am running windows 7 on a sage dedicated server and am using a program called prio. I set the priorities for all of the programs running. By setting the io and cpu priorities low for other programs I extended the time I ran out of memory to the current once every few months.

Lance

On Tue, May 16, 2017 at 6:44 AM, Joseph Shuttlesworth < notifications@github.com> wrote:

The additional memory usage is only while Carny is running, but just about anything you do with any recording will cause it to run again, so the memory might as well be considered always in use. Also you can turn the maps off and still get multi-threaded performance, that's close to linear per core dedicated to the task. I have been looking at more efficient memory allocation strategies for the Show objects.

For my own curiosity, I packed the Show object into individual efficient off heap ByteBuffers and with the maps the total heap size was 354MB (vs 420MB). This of course means that any time you want to do anything with a show, the part that you want needs to be converted back into an object which does create a GC situation, but the reality is that you're not looking at 100% of the shows in the database at any time, so why do we always have all of the data always loaded? When I tried this, I was careful to maintain the identity of the objects since SageTV relies heavily on it, just all of the internals were always being re-created/retrieved. In the UI, I was unable to perceive any significant delays due to the way the data was being loaded.

For what it's worth, I do a lot of my testing on a Windows 7 box. The standard SageTV setup is actually fairly light on memory per client connection. It's definitly the graphic rich UI's causing many of the problems we see. Also I left the Comskip plugin running once while I was doing profiling and it is constantly opening and closing it's own configuration file which creates a not insignificant amount of GC on it's own.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/287#issuecomment-301744492, or mute the thread https://github.com/notifications/unsubscribe-auth/AbW4gIAVX4pKXMrViVYs0Lm_3hxzA7zLks5r6X33gaJpZM4NbwvK .

-- "The only thing necessary for the triumph of evil, is for good men to do nothing." - Edmund Burke 1729 – 1797

[enternoescape]

@Narflex I've been running my production server on 3 weeks for about a week now and I haven't seen any adverse problems. If everyone is ok with this, I'll make the STV change and submit it.

[Narflex]

Go for the STV change, no problem there.

Jeff Kardatzke Sent from my Android

On May 26, 2017 7:02 PM, "Joseph Shuttlesworth" notifications@github.com wrote:

@Narflex https://github.com/narflex I've been running my production server on 3 weeks for about a week now and I haven't seen any adverse problems. If everyone is ok with this, I'll make the STV change and submit it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/287#issuecomment-304419491, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDBZ6NhzyL7Dxllb7y9Q985Cl-4eAks5r94QsgaJpZM4NbwvK .