Improve performance by using concurrent.futures

[bloodnighttw]

Recently I want to use this script for my personal project, then I found that if I don't clone meta-upstream to upstream/, it will take a lot of time to generate data when using ./update.sh.

What I do

This is just an example.

I just turn

...
for it in index:
    print(f"Processing {component} {it['version']} ")
    jar_maven_url = get_maven_url(
        it["maven"], "https://maven.fabricmc.net/", ".jar"
    )
    compute_jar_file(
        os.path.join(UPSTREAM_DIR, JARS_DIR, transform_maven_key(it["maven"])),
        jar_maven_url,
    )
...

into

...
def compute_jar_file_concurrent(component, it):
    print(f"Processing {component} {it['version']} ")
    jar_maven_url = get_maven_url(
        it["maven"], "https://maven.fabricmc.net/", ".jar"
    )
    compute_jar_file(
        os.path.join(UPSTREAM_DIR, JARS_DIR, transform_maven_key(it["maven"])),
        jar_maven_url,
    )
    print(f"Processing {component} {it['version']} Done")
...

...
with concurrent.futures.ThreadPoolExecutor() as executor:
    for it in index:
        executor.submit(compute_jar_file_concurrent, component, it)
...

Comparison

Here is the time comparison about what the code what I change. (Left side is my code, Right side is the original code.)

Here is my test enviroment information:

OS: Arch Linux x86_64 
Kernel: 6.9.1-zen1-2-zen 
Uptime: 2 hours, 35 mins 
Terminal: kitty 
CPU: 12th Gen Intel i5-12400 (12) @ 4.400GHz 
GPU: NVIDIA GeForce RTX 3060 
GPU: Intel Alder Lake-S GT1 [UHD Graphics 730] 
Memory: 10164MiB / 31837MiB 

#

Note

I have noticed some order change in launcher/net.fabricmc.intermediary/index.json,these also change sha in launcher/index.json. I just need to figure out what happened in these change.

[Scrumplex]

I used to use Python multiprocessing for tasks like this. My only concern with this codebase was that some functions rely on global variables, which make it hard to parallelize

[bloodnighttw]

I used to use Python multiprocessing for tasks like this. My only concern with this codebase was that some function rely on global variables, which make it hard to parallelize

I think I don't touch any global variable, but I will pay attention to it.

[bloodnighttw]

After a day of debugging, I decided not to commit the code that optimize the forge update code. Because it cause so many bug, and the forge/neoforge update code is a little bit complex to me to make it parallelize,sorry about that.

In my pc, my branch script output is same as the original script. (I also found that the original script can't have same output with PrismLauncher/meta-launcher in my PC,I don't know why.)

The code is ready to review and test,thx.