search

flipdazed / weather-modelling

Deep Architectures for Weather Modelling
4 stars 1 forks source link

Parse Live Data from Cloud EC2 #8

Open flipdazed opened 7 years ago

flipdazed commented 7 years ago

Aim Need a method of pull live data from the cloud in Amazon EC2

flipdazed commented 7 years ago

HDF5?

http://stackoverflow.com/questions/16628329/hdf5-concurrency-compression-i-o-performance/16629243#16629243

tested hdf5 against cPickle

import h5py
import numpy as np
import timeit
import cPickle as pkl

a = np.random.random((10000,10000))

t1 = timeit.default_timer()
with open('data.pkl', 'wb') as f:
    pkl.dump(a,f)
t1 = timeit.default_timer() - t1

t2 = timeit.default_timer()
h5f = h5py.File('data.hdf5','w')
h5f.create_dataset('dataset_1', data=a)
h5f.close()
t2 = timeit.default_timer() - t2

print 'cPickle: {} secs'.format(t1)
print 'hdf5: {} secs'.format(t2)

## -- End pasted text --
cPickle: 57.8580451012 secs
hdf5: 1.47937297821 secs
flipdazed commented 7 years ago

Can set up an HDF5 server: https://github.com/HDFGroup/h5serv