Utilize multiprocessing in step 3 to dramatically improve performance

[antiguadev]

Currently step3 is the slowest step of the process. It is also single threaded. However, I see no reason why this step cannot take advantage of multiprocessing to utilize the full performance of the CPU and build many GIFs simultaneously.

As a test, I tried to modify the code myself and was able to build 16 GIFs at a time with my 16 thread CPU. I'll share what I did here, however I am an incredibly novice "developer" (I hesitate to even call myself that) and no prior experience in Python.

Added the following to build.py

import multiprocessing

def generate_gif(filename: str):
    if filename.endswith(".png"):
        print(f"Converting spritesheet to gif for {filename}")
        crop_and_save(
            filename,
        )
        fps = 5
        convert_pngs_to_gif(filename, fps)

def generate_all_gifs(filename: str):
    with multiprocessing.Pool() as pool:
        pool.map(generate_gif, filename)

Note the "fps = 5". I could not figure out how to get the fps variable passed on through to generate_gif (like I said, novice). So I just stuck it in there like that until I can figure out the proper way to do it.

modified main to:

def main():
    print("Starting step 3: Converting sprite sheets to gifs")

    for folder in [output_gifs_directory, output_images_directory, temp_directory]:
        setup_directory(folder)

    generate_all_gifs(sorted(os.listdir(input_directory), key=sort_function))

[jalagar]

This, is, AWESOME! Yes multiprocessing is a great idea (compared to multithreading which doesn't work in Python given the GIL - global interpreter lock).

Do you want to try to create a PR and merge the code? I can add comments to the PR to how to pass in fps, but that way you'll get credit for the code.

If you aren't too familiar, you can follow the instructions here https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request.

Anyways let me know so I can add you as a collaborator.

[antiguadev]

Ok give me some time to improve my code a bit more, it's a bit janky and the terminal output isn't ideal. I'd be happy to do a PR and get credited.

[jalagar]

Great! I just added you as a collaborator

[jalagar]

Hey @antiguadev any update here? Some people were looking for speeding up step3 :)

[antiguadev]

Hey man so sorry but I haven't found the time. If anyone else is willing to implement my suggestion please go ahead as I am unable to commit (lol) to getting this done any time soon.

[jalagar]

Hey no problem @antiguadev I was able to use your code and modify it for step1 and step3. You can take a look at the PR here: https://github.com/jalagar/Generative_Gif_Engine/pull/67. Feel free to add any questions or ask anything.

Thank you for the starter code and idea! I used pool.starmap to feed the multiple arguments in and it all worked. It improved step1 and step3 by 4-8x!