search

Rob174 / detection_nappe_hydrocarbures_IMT_cefrem

0 stars 0 forks source link

Test read speed of the raster images versus hdf5 file #7

Closed Rob174 closed 3 years ago

Rob174 commented 3 years ago

What is the time required to open one raster image versus to open it if it is in a hdf5 file ?

Rob174 commented 3 years ago 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ Name                        ┃ Access time (ms)  ┃ Shape of the image ┃ Memory size (MB)  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ 027481_0319CB_0EB7 avg time │ 834.939632        │ (10600, 18441)     │ 781.8983999999999 │
│ 016505_01F10F_CE84 avg time │ 876.056842        │ (10655, 18706)     │ 797.2497199999999 │
│ 016753_01F88A_4864 avg time │ 970.7034199999999 │ (10648, 18759)     │ 798.9833279999999 │
│ Global avg time             │ 893.8999646666665 │ -                  │                   │
└─────────────────────────────┴───────────────────┴────────────────────┴───────────────────┘
Rob174 commented 3 years ago

and with h5py:

┌───────────────┬───────────────┬──────────────┬───────────────┬──────────────┐
│ Name          │ Access time   │ Access time  │ Shape of the  │ Memory size  │
│               │ write (ms)    │ read (ns)    │ image         │ (MB)         │
├───────────────┼───────────────┼──────────────┼───────────────┼──────────────┤
│ 027481_0319C… │ 585.144       │ 0.0          │ (10600,       │ 781.8983999… │
│ avg time      │               │              │ 18441)        │              │
│ 016505_01F10… │ 599.21039999… │ 0.0          │ (10655,       │ 797.2497199… │
│ avg time      │               │              │ 18706)        │              │
│ 016753_01F88… │ 571.9643      │ 998400.0     │ (10648,       │ 798.9833279… │
│ avg time      │               │              │ 18759)        │              │
│ Global avg    │ 585.43956666… │ 332800.0     │ -             │              │
│ time          │               │              │               │              │
└───────────────┴───────────────┴──────────────┴───────────────┴──────────────┘

The read speed is most of the time less than 1ns (2.3 GHz processor) which is a bit too quick. Problem with the method to get the read speed (difference between time in ns) ?

We can take as an approximation 1ms as it is the upper bound of the values

Rob174 commented 3 years ago

So, h5py is quicker