issue with starmap outputs

[ricomnl]

I have a modal function (https://github.com/whitead/minimalaf/blob/main/af.py) that returns a zipped directory as bytes and i'm calling the function in parallel with starmap, then saving the zip file:

i = 0
for res in af.starmap(input_list, order_outputs=True):
    with open(f"{output_folder}/{i}.zip", "wb") as f:
        f.write(res)
    with zipfile.ZipFile(f"{output_folder}/{i}.zip","r") as zip_ref:
        zip_ref.extractall(f"{output_folder}/{i}")
    os.remove(f"{output_folder}/{i}.zip")
    i += 1

I'm noticing that it sometimes saves teh same directory content multiple times and initially thought it had something to do with order_outputs but it didn't change anything when I set it to true - why could this be?

[ricomnl]

the same happens when I do:

i = 0
for seq_name, seq in binder_dict.items():
        funcs.append(af.spawn(seq_name, seq, num_models=num_models, num_recycles=num_recycles, num_seeds=num_seeds))
    for res in modal.functions.gather(*funcs):
        with open(f"{output_folder}/binder_{i}.zip", "wb") as f:
            f.write(res)
    i += 1

[mwaskom]

Hey @ricomnl if you're seeing the same issue when using starmap or spawn it suggests to me that it might be coming from your data not Modal, since those go through different pathways. How do you know the results are guaranteed to be unique?

[mwaskom]

Feel free to reach out on modal.com/slack if you're still having issues!