Use LRU caching for all pixmaps

[kiddac]

https://github.com/OpenViX/enigma2/commit/df56a9ccc9c3a5fe4cf417c1fc80f52de76ec8b3

You have wrongly made the assumption that all pixmaps are static and never change. This is not the case. This caching of pixmaps has now broken all my skins that bring in cover art for channels as they use the same png name. To fix this I am now providing all my users a copy of the OpenATV LoadPixmap.py.

Can you please undo this LRU caching for all pixmaps, there was nothing wrong with the original LoadPixmap.py Thanks.

[RobvanderDoes]

Who are 'my users'?

[kiddac]

The OpenVix users of my skins, that are now finding things broken because of this change.

[ViX-Sicilian]

No one revert anything at the moment, not until current release build completes.

[RobvanderDoes]

What are 'your skins'?

[kiddac]

The problem is that myself and people like Chab's bring in image scrapes in their skins, that bring cover art into certain screens. These multiple images use the same png name. i.e tile1.png, tile2.png. etc

These are now being cached unless the gui is restarted to refresh them, so they are not changing at a set time interval as expected.

The commit mentioned cache unless specifically disabled. What is the alternative of how to disable caching in my python component???

[RobvanderDoes]

And who is 'Chab'?

[kiddac]

Next you will be asking who is Rossi, Who is Digiteng?

[Huevos]

@kiddac, if the "cache" is displaying the wrong images the commit will be reverted. Please be patient as we are halfway through a rebuild.

[ViX-Sicilian]

We will have to wait after this build completes please. I’ll do another release build in day or two after completes.

[kiddac]

Thank you. It's not displaying the wrong images as such. The component is doing its job - Caching the images. But some of us obviously lesser known skinners :)) do funky things with changing the images for visual eye candy . So we do not need the caching of all pixmaps, or at the very least need a solution to get around the caching in our custom components.

[SimonCapewell]

@kiddac Are these image names outside of the names referred to in the OpenVix code or are they used in the additional custom components you distribute with your skins? I was looking at adding static caching in EPGList.py for all the little graphics used in the grid to try to get it loading more snappily, but would have to shelve that idea if this breaks stuff.

[kiddac]

They are just pngs/jpgs brought in with ePixmap for example. In this example on the menu screen.

`

    <ePixmap pixmap="q-toppicks/toppicks2.jpg" position="806,386" size="246,327" backgroundColor="mid" transparent="0" zPosition="2" />

    <ePixmap pixmap="q-toppicks/toppicks3.jpg" position="1070,386" size="246,327" backgroundColor="mid" transparent="0" zPosition="2" />

    <ePixmap pixmap="q-toppicks/toppicks4.jpg" position="1334,386" size="246,327" backgroundColor="mid" transparent="0" zPosition="2" />

    <ePixmap pixmap="q-toppicks/toppicks5.jpg" position="1598,386" size="246,327" backgroundColor="mid" transparent="0" zPosition="2" />

    <ePixmap pixmap="q-toppicks/toppicks6.jpg" position="542,780" size="240,320" backgroundColor="mid" transparent="0" zPosition="2" />

    <ePixmap pixmap="q-toppicks/toppicks7.jpg" position="806,780" size="240,320" backgroundColor="mid" transparent="0" zPosition="2" />

    <ePixmap pixmap="q-toppicks/toppicks8.jpg" position="1070,780" size="240,320" backgroundColor="mid" transparent="0" zPosition="2" />

    <ePixmap pixmap="q-toppicks/toppicks9.jpg" position="1334,780" size="240,320" backgroundColor="mid" transparent="0" zPosition="2" />

    <ePixmap pixmap="q-toppicks/toppicks10.jpg" position="1598,780" size="240,320" backgroundColor="mid" transparent="0" zPosition="2" />`

but these images get changed via a custom python componet and a cron script. But as they are now cached all the time. They no longer change.

[Huevos]

@SimonCapewell I tried caching with picons a while ago and didn't see any performance gain. Rob suggests there is a performance benefit for older receivers such as the Ultimo with this code. But that is academic if it leads to non-static images not displaying correctly.

Personally I would rather avoid caching altogether and any pull requests that contain it.

[IanSav]

If you look at the LoadPixmap source you will find that you can turn the caching off on demand:

def LoadPixmap(path, desktop=None, cached=None):
    if cached is None or cached:
      ret = _cached_load(path, desktop)
      return ret
  else:
    return _load(path, desktop)`

Adding a "cached=False" to your LoadPixmap commands should suppress the caching.

[Huevos]

Ian, that is just another example of the tail wagging the dog.

[kiddac]

Yes but the image is loaded via skin code epixmap (which uses loadpixmap.py. My images are only changed in the original location, but has the same file name. So as far as I am aware you cannot put cache="false" on the epixmap in the skin. Correct.

[IanSav]

Huevos, I look at it as using the options as always provided but only recently reactivated. The caching intention was apparently coded about 11 years ago though it wasn't reactivated until recently. The "cache" argument has been around for a VERY long time and is included in OpenPLi, OpenViX, OpenATV and Beyonwiz. (I only checked those four images.)

Remember that I am not behind this change. I can say that it has improved UI performance in the Beyonwiz image, from where this code was cherry picked.

[Huevos]

Ok, but it needs to be accessible from the skin then.

[Huevos]

Then all parties would be happy.

[IanSav]

Kiddac, I suspect that you are correct. The name, not the content is the key to the cache so this may still be a problem for you.

There are simple ways that this could be addressed, like checking some inode parameters to see if the file content has changed (this would have a lesser performance but still better than not caching at all), but perhaps this is beyond the scope of this issue.

[IanSav]

Huevos, if the caching algorithm is tweaked then the caching could be fully automatic and transparent. No skin changes would be required.

[SimonCapewell]

Just in case anyone is in any doubt about performance benefits of caching, I've measured a 100ms load time on my Xtrend 8500 for the ~50 pngs required by the graph EPGList code without caching. Total load time for the screen tends to be around 800ms IIRC, which is long enough to wonder if the machine saw your button press. With this caching, it drops to around 20ms.

[RobvanderDoes]

So it comes down to someone willing/able to do the tweaking IanSav suggested....

[SimonCapewell]

I'll have a look over the weekend. The algorithm will be to checking the image's updated time against the one stored in the cache and the file is newer, refresh the image in the cache.

[andyblac]

not sure how the cache works, but it would be better to make a checksum of the file in cache and compare that to the local file if it does not match then update the cache, rather than just going off date/time.

[SimonCapewell]

Hmm, wouldn't that require loading the contents of the file to compare the checksums each time?

[andyblac]

you could create a cache file for the checksums and compare to that ?, so as loadpng loads the file is creates the checksum and stores it with location, then on next load it check if a checksum exists and uses it if available if no checksum it just loads the file.

just a thought, besides think it would still be quicker checking the checksum than loading a png

[SimonCapewell]

If it's a CPU bound issue, then turning the content into a png object will be the bottleneck and the checksum approach will be good. If it's IO bound then checksum won't be such a good idea. In my normal line of work, my money would be on IO being the bottleneck, but this is slightly different class of hardware! Best think is to profile all 3 approaches (updated time, checksum, no caching) and make a decision based on the result.

[andyblac]

do some tests, but from my past experience and guess it is a CPU issue, as the graphics is (from what I remember, but this was before ARM) was all done in the CPU cycles, I don't think flash ie I/O is the issue, after all we're only talking a few K file sizes.

[IanSav]

Processing the image files in any way will defeat the cache and the purpose of caching the files in the first place.

If the tests can be based on only data available in the inode then this will come from RAM as the inodes are usually in system cache. The cost to query an inode value should be SIGNIFICANTLY less than finding and accessing the actual file in the filesystem (for example the find, open, process and close time for accessing the actual file). Times, inode number, etc could all be a basis for validating the image cache and either using the cache image or refreshing the cache.

In Kiddac's case his fancy displays should refresh much faster and consume less system resources while a image is stable and cached. With the proposed cache validation the image caches should all reset and reload the images when the cron job swaps the images around. They will then reload faster until the next cron loop.

[ViX-Sicilian]

Guys, what do we do with this? Is this something that can be resolved in the near future or do we revert this this until resolved? I'm bumping DEV builds later today so would like to make a decision.

[RobvanderDoes]

There is no hurry. The only person complaining about this, has a work-around.

[IanSav]

I can't work on it in the near future. Perhaps SimonCapewell will be able to complete his evaluation over the weekend as he proposed.

[kiddac]

There is no hurry. The only person complaining about this, has a work-around.

I only have a work around when people say how do I fix this in your skin and then I have to manually give them the old loadpixmap.py to load onto their box.

But I assume this is going to be a problem for any image that is used a placeholder and then that image is changed in the background.

I haven't tested this, but this might effect things like the IMDB plugin that brings in cover art, it will most certainly effect iptv plugins that download picon art in realtime.

It will probably effect dreamplex plugin that downloads backdrops, cover art, music art. Etc.

other things that might need testing off the top of my head. Picture player. Select skin... thumbnail.

So I don't think its just a case of just affecting mine and a handful of other skins.

[IanSav]

You could use the "cache=False" option in your LoadPixmap code to suppress the caching.

From memory IMDb is working fine with the cache code in place. I don't think it uses LoadPixmap.

[kiddac]

IanSav we are talking about images in skin screens <ePixmap pixmap="blahblah".... That use loadpixmap.py via the code in skin.py.

[IanSav]

You mentioned the poster image in IMDb.

This is the skin widget

<widget name="poster" position="4,90" size="96,140" alphatest="on" />

Here is the code that creates that widget:

self["poster"] = Pixmap()
self.picload = ePicLoad()
self.picload.PictureData.get().append(self.paintPosterPixmapCB)

There is some screen scraper code and an image fetch from IMDb to get the poster image.

And here is the code that changes the image:

def paintPosterPixmapCB(self, picInfo=None):
    ptr = self.picload.getData()
    if ptr != None:
        self["poster"].instance.setPixmap(ptr)
        self["poster"].show()

As you will note the LoadPixmap.py code is not used.

[RobvanderDoes]

I never observed any issues in IMDB.

[kiddac]

I am not on my box at the moment. Was just thinking of plugins that could be potentially affected. Not checked any of the code of anything at the moment

[kiddac]

Just to add further confusion to this. I have just installed openvix's version of loadpixmap.py and LRUCache.py to my openatv build. (I don't have a vix compatible box) Despite copying over the python components my images still quite happily change. Yet several people are telling me they no longer change after they updated there openvix.

Checking the individual python modules, I do not have "from thread import RLock" on my OpenAtv. Is this needed for the cache of the files to happen.

Because at the moment I am unable to do any further testing on this, and just relying on people telling me what is broken after this component update.

[RobvanderDoes]

I think we should not be chasing 'hear say' (or ghost) issues.

[kiddac]

OK, everyone is lying. Thanks for breaking something that wasn't broken in the first place. Have a nice day.

[ViX-Sicilian]

@ Rob, this is a genuine issue that i've seen reported on a forum by a couple of experienced E2 users.

If a resolution is not found I will be reverting this.

[IanSav]

@Kiddac, without seeing your code it is difficult to theorise an explanation for the issue reports. Given that all the problems started when the cache code was reactivated I would suggest that we give SimonCapewell a little time to complete his research and perhaps he can improve the cache and the whole issue can be addressed by the end of this weekend.

[SimonCapewell]

@kiddac Most likely you're running your box with more graphics loaded than some of your users so you've got cache rotation refreshing the images. This would also explain why some people are having problems and others aren't. If I'm understanding the cache code correctly, there are 256 entries, which given the use of 50 entries by just the grid EPG, you can easily see more than 256 individual graphics getting used up in a graphics heavy skin.

[AbuBaniaz]

Issue is not specific to EPG, even in the main menu it is manifest according to these posts. https://linuxsat-support.com/thread/134771-skin-support-post-questions-for-all-my-skins-in-here/?postID=476099#post476099

He doesn't use ViX, might be an idea for those looking at the issue to load his skins.

[RobvanderDoes]

OK, everyone is lying.

I have no idea about that. We have a policy of reproducing issues before looking into possible solutions. We don't know about these skins, we don't have them on our feeds and we can't possibly prevent or solve issues with all kinds of exotic plugins. Please feel happy about the fact that you have found a simple a work-around, and that in spite of the above mentioned arguments already two of our team members have offered help to look into this.

[kiddac]

I appreciate those people looking into my concerns regarding this update and understanding the potential issues, while others are dismissive of it. And just because openvix skins are very standard, with the only images being backgrounds, selectbars and icons, it doesn't mean the issue I have raised isn't a potential problem for other skins and plugins. As for saying we don't know about these skins, people like IanSat, Huevos, AbuBaniaz are fully aware of mine and Chab's skins presense as they are some of the post popular skins used in the UK. So this isn't a problem only effecting a handful of people, it's potentially effecting a lot of users of OpenVix.

And I will reiterate. Making people load an alternative loadpixmap.py onto their box or me including it in my IPKs is not a solution. It is an unnecessary workaround.

[Huevos]

Yes, we are very accommodating even to people that have given us plenty of stick in the past.

[kiddac]

You call it stick, I call it constructive critism by highlighting issues that have been overlooked. All my arguments in the past have been valid arguments. But that is a different conversation. Unfortunately as I am not a user of openvix I tend to find out about these issues late in the process from users of my skins. So I fully understand people getting frustrated when they have put in all the hard work to change things for what they think is for the better, only for issues and concerns to be raised late in the game. But my points are always valid and legit.