CubeBrowser / cube-explorer

browser based exploration of iris cubes
BSD 3-Clause "New" or "Revised" License
8 stars 12 forks source link

Speed up creation of HoloViews objects #29

Closed philippjfr closed 8 years ago

philippjfr commented 8 years ago

In the past HoloViews has generally been used for reasonably small datasets, where loading or generating the data is many times more expensive than creating param.Parameterized objects. However in this project we'll be working with iris cubes, which can potentially be very large and can easily be grouped into tens or even hundreds of thousands of individual Images, Contours or other Element types. At that point creating this many objects, which may each also create other parameterized objects, can become a serious bottleneck.

param was designed with the idea that the user should be able to tweak each parameter on each object, when declaring many thousands of objects at once however it is clear that the user will not tweak each individual parameter and it might indeed be desirable to share certain parameters across all these objects. Therefore there is this tension between being able to tweak parameters and quickly declaring thousands of identical objects.

This issue is meant as a place to brainstorm how we can speed up creation of objects in HoloViews, by providing better control over when Parameterized objects make deep copies of their parameters. This could take the form of a context manager, which could be used in any place when a lot of identical objects will be generated, an option on Parameterized objects or something else entirely.

I've already opened https://github.com/ioam/param/pull/122, which speeds up object creation by ~50%, such that we can create 20k Image objects in about 5 seconds. Hopefully we can reduce that further by coming up with a good proposal here.

marqh commented 8 years ago

param optimisation will address this, closing for this project