search

diatomic / tess2

Parallel Delaunay and Voronoi Tessellation and Density Estimation
Other
27 stars 7 forks source link

tess_save() should work when not all blocks fit in memory #3

Closed mrzv closed 9 years ago

mrzv commented 9 years ago

Issue by Dmitriy Morozov Saturday Nov 22, 2014 at 21:50 GMT

Much of the logic can be copied from diy/io/block.hpp, but I'm not sure how it would interact with pNetCDF.

mrzv commented 9 years ago

Comment by Tom Peterka Saturday Nov 22, 2014 at 22:37 GMT

It might be time to ditch pnetcdf for tess2. We would need to redo the reader for draw and dense. Dense is a diy1 program that I need to convert to diy2, but draw is serial. We would need a standalone serial reader that reads all blocks into one process, or rewrite draw as a one-process diy2 program.

mrzv commented 9 years ago

Comment by Dmitriy Morozov Saturday Nov 22, 2014 at 23:24 GMT

I think rewriting draw as a one-process diy2 program makes the most sense. We'd just have to tell it to use a single thread, rather than all available cores (-1). Moreover, if you don't want to deal with diy's foreach() and such, you could just cycle through the blocks yourself, accessing them through Master::block(i) or even Master::get(i). In the latter case, you'd be able to limit the number of blocks in memory (not that you'd ever want to with draw).

mrzv commented 9 years ago

Comment by Dmitriy Morozov Friday Feb 20, 2015 at 19:57 GMT

Switching to diy::io::{read,write}_blocks() in 7fabf16 fixed this issue.