letmaik / rawpy

📷 RAW image processing for Python, a wrapper for libraw
https://pypi.python.org/pypi/rawpy
MIT License
587 stars 67 forks source link

(cont'd) Remove gil from C implementations to enable reading raw files in parallel threads #220

Closed jontyrudman closed 3 months ago

jontyrudman commented 6 months ago

Apologies for the radio silence; I do intend to get this done, I just need to work out the best way of testing that two threads have successfully run in parallel.

This work was started by kmaddock, new PR made because they were unresponsive.

jontyrudman commented 6 months ago

Just rebased to fix those CI issues for Python 3.12 on macos and windows.

Struggling to find a concrete way to test that the nogil stuff is working - I know it does because I've seen the benefits in my application but I can't get parallel imread and postprocess to look tangibly different from ones in series in the test func I'm working on right now.

EmersonDove commented 4 months ago

This worked super well for me first try, this code below now runs correctly where the main thread now correctly prints 10 times a second while the reading thread runs:

import threading
import time
from pathlib import Path

import rawpy
from PIL import Image as PIL_Im

def process_rawpy(path):
    with rawpy.imread(str(path)) as raw:
        rgb = raw.postprocess(use_camera_wb=True)

    return rgb

def open_image(path: str) -> PIL_Im:
    while True:
        if isinstance(path, str):
            path = Path(path)

        try:
            result = process_rawpy(path)
        except Exception as e:
            print(f'Failed to open raw image {path} with rawpy, falling back to Pillow: {e}')

if __name__ == "__main__":
    threading.Thread(target=open_image, args=('/path/to/arw',)).start()

    while True:
        print("Main Loop")
        time.sleep(0.1)

Would be great to get this merged, going to make my own pypi distribution for it for our own usage so having it in mainline would be great.

letmaik commented 3 months ago

OK, let's merge this, I think we got enough evidence that it works.