Benchmark 1.25B pixel image

[dchaley]

Follow-up to #163, do the same thing with 1.25B pixels. Dimensions: 36000*35000

import numpy as np
import smart_open
import timeit

path = 'gs://davids-genomics-data-public/cellular-segmentation/hubmap/fake/hbm873.pcpz.247.quadtile.2670M-px/input_channels.npz'

t = timeit.default_timer()
with smart_open.open(path, "rb") as input_channel_file:
    with np.load(input_channel_file) as loader:
        # An array of shape [height, width, channel] containing intensity of nuclear & membrane channels
        input_channels = loader["input_channels"]
input_load_time_s = timeit.default_timer() - t

subselection = input_channels[500:36500, 1000:36000]

output_path = 'gs://davids-genomics-data-public/cellular-segmentation/hubmap/fake/hbm873.pcpz.247.quadtile.1260M-px/input_channels.npz'

with smart_open.open(output_path, "wb") as output_file:
    np.savez_compressed(output_file, input_channels=subselection)

[dchaley]

Image created & currently benchmarking.

[dchaley]

Still running, at least 16 hours later. Well, at least it didn't crash…

[dchaley]

I killed it last week after waiting ~2 days– it racked up quite the bill 😰 Not sure how to record this in our results… cc @WeihaoGe1009

[dchaley]

We recorded it as a failure, with no information on time.