search

borevitzlab / timestreamlib

DEPRECATED. Please use the current version of the TimeStream tools at https://gitlab.com/appf-anu/pyts2.
https://gitlab.com/appf-anu/pyts2
GNU General Public License v3.0
7 stars 4 forks source link

Are we using timestream/manipulate/netcdf.py? #100

Closed Joelgranados closed 9 years ago

Joelgranados commented 9 years ago

The function ts_to_tsnc is not being used anywhere.

  1. What are the plans for this file?
  2. what is the idea with keeping the timestream in one big 4-D cube?
  3. How much memory does this actually need to run? The pixels value keeps all the images of the timestream in memory, how much memory does this actually use with 2000 3Mb image files?
kdm9 commented 9 years ago

1) The "version 2" of the timestream spec was going to be based on NetCDF. This script was to convert between the two. Ask @TimeScience about the status of these plans. 2) Ease of slicing, basically. In all dimensions. See THREDDS and some of the stuff out of NASA/JPL 3) It writes them to disk transparently, from memory. I've made ~500GB ones on the 32GB RAM server. Think it peaked at ~10 GB RAM. Not ideal, but it has not been optimised at all.

kdm9 commented 9 years ago

3, addendum) I think it uses mmap??

Joelgranados commented 9 years ago

We are not really using netcdf and it is a considerable overhead when installing on windows. I'm going to remove it from the requirements.

Also, Going to leave this open until we decide what to do with the rest of the netcdf code.

Joelgranados commented 9 years ago

ooops, nevermind, Kevin beat me to it :)