[WIP] VTK CUDS Memory saving

[stefanoborini]

Tentative implementation for #35. I decided not to copy the CUDS into a CUDS with less data. Instead, this operation is performed when the VTKCUDS are populated. It saves one data copy.

[stefanoborini]

@itziakos needs testing, but this is the proof of concept.

[itziakos]

The restriction should happen at the CUBADataAccumulator when the data are extracted from the cuds object. Without that you still get a spike in memory usage. In sort by the time the load_data_onto_vtk has been called we already have extracted the data attributes from the CUDS objects and they take space

[itziakos]

How do you know which attributes are available? The Mayavi source needs to know that, so that we can provide a pull down for the user to select

[itziakos]

Apart from the comments above I think that the implementation is in the right direction. I would suggest however to make a new CUDSSource for example SlimCUDSSource. it would be nice to have both available since both paradigms are useful. If you have memory the CUDSSource will be the best to use for speed, while the SlimCUDSSource will be optimized for low memory usage.

[kitchoi]

The restriction should happen at the CUBADataAccumulator. Without that you still get a spike in memory usage.

I see that there are two levels of memory usage/saving here: (1) CubaDataAccumulator: like you said here if the restriction happens in CubaDataAccumulator.init by defining keys (2) the tvtk Array Objects: which is saved by passing keys in load_onto_vtk as is implemented here

What do we want?: Say the Particles have point_scalars "TEMPERATURE" and "MASS"

cuds = Particles("some_name", data = dict(TEMPERATURE..., MASS=...))

I suppose you want to load nothing to the tvtk.PointData obj here? Or the first data attribute as DEnum would do so for you?

source = SlimCUDSSource(cuds=cuds)
source._vtk_cuds.particle_data.keys  # <--- what should this be?

Need to run for swimming lesson :(

[itziakos]

Please note

What do we want?: Say the Particles have point_scalars "TEMPERATURE" and "MASS"

cuds = Particles("some_name", data = dict(TEMPERATURE..., MASS=...))

This is not how CUDS containers behave the data attribute is a property of the container it does not reflect what is stored in the data attribute of the particle and bond items inside the container.

I guess you mean that the Particle container contains particle and bonds items whose data attributes contain TEMPERATURE and MASS. I just wanted to make sure that we are on the same page

[kitchoi]

I guess you mean that the Particle container contains particle and bonds items whose data attributes contain TEMPERATURE and MASS. I just wanted to make sure that we are on the same page

Yes that's what I meant. I am sorry. Thanks for correcting this. I should comment when I am not in a rush...

(1) CubaDataAccumulator: like you said here if the restriction happens in CubaDataAccumulator.init by defining keys

And I should correct for this as well: I meant by appending the data to CubaDataAccumulator on demand.

[stefanoborini]

Ok, I assumed that the only place we could actually save memory was on the vtk object. I don't know the rest of the source and the involved objects so I left them untouched, but if we can push it further down it's probably a bigger save.

Tomorrow I'll scout if it's possible and doesn't introduce unexpected problems.

[stefanoborini]

Retiring this PR. After running in circles and finding obstacles here and there, I settled for another solution which is being opened later.

[stefanoborini]

Keeping the branch alive for now, but just in case.