CodeSchmiedeHGW / BLITZ

Bulk Loading and Interactive Time series Zonal analysis
GNU General Public License v3.0
1 stars 0 forks source link

General Performance #34

Closed PiMaV closed 1 week ago

PiMaV commented 2 weeks ago

Performance discussion: https://pyqtgraph.readthedocs.io/en/latest/api_reference/graphicsItems/imageitem.html#performance "Row-major" was actually used in the early versions. It probably got lost along the way... But this turns images 90° (flips x&y); this shouldnt change our program too much(as we also have transpose in there as well) but this would mean, we have to flip the horz & vert line plots as well.

PiMaV commented 2 weeks ago

I've made a small example to bench both and dont find any difference:

for 10, 10000,5000: Average image loading time with row-major order: 0.7376 seconds Average image loading time with col-major order: 0.7344 seconds

for 4,10000,10000: Average image loading time with row-major order: 18.2147 seconds Average image loading time with col-major order: 18.2413 seconds

import numpy as np
import pyqtgraph as pg
import time

def benchmark_image_loading(imageAxisOrder, repetitions=10, image_size=(5000, 1000)):
    pg.setConfigOptions(imageAxisOrder=imageAxisOrder)

    times = []
    for _ in range(repetitions):
        start_time = time.time()

        # Generate large image data
        data = np.random.normal(size=image_size)
        data[1000:3000, 1000:3000] += 2.
        data = pg.gaussianFilter(data, (3, 3))
        data += np.random.normal(size=image_size) * 0.1

        # Process image data
        img = pg.ImageItem()
        img.setImage(data)

        processing_time = time.time() - start_time
        times.append(processing_time)

    avg_time = sum(times) / repetitions
    print(f"Average image loading time with {imageAxisOrder} order: {avg_time:.4f} seconds")
    return avg_time

# Benchmark with row-major and col-major configurations
row_major_time = benchmark_image_loading('row-major')
col_major_time = benchmark_image_loading('col-major')
PiMaV commented 2 weeks ago

I've implemented and trying in BLITZ with a 3.3 GB Dataset: Size: 2500x300 px Number of Images : 4500 "Loading", "STD" and "normalize with STD" on my machine took: 12, 11, 28 [s] for row-major and NOT row-major its below 0.4 s difference

-> You can doublecheck and/or Close the issue

irkri commented 2 weeks ago

Its understandable that it makes no big difference. We do all the heavy calculations like PCA on numpy arrays, not on the ImageData class. Everything that pyqtgraph does is displaying the images (and probably all operations that appear when right-clicking on the image). Switching everything to row-major would just speed up some examples where there are a lot more columns than rows (or the opposite way around). I think its not necessary to do anything regarding this.

PiMaV commented 1 week ago

Double checked -> Does not make sense. -> Done