High Memory Usage Leading to System Crash During `convert visium` Command

[jkbenotmane]

Environment:

• scResolve version: 0.1.0
• Operating System: Ubuntu 22.04
• Python version: Python 3.9
• Intel Cor i9, 64GB DDR5 + 66GB SWAP RAM
• Please check the installation script, as I had version issues with the installed matplotlib as well as some imports

• Description: I am encountering a significant memory usage issue when running the scresolve convert visium command as part of a loop in my script. The loop processes multiple samples, but it leads to excessive RAM consumption before completion of and crashes my system before reaching any output after: "[2023-12-26 14:08:31,700] INFO : Running scresolve version 0.1.0". Below is the loop that leads to the memory crash:

for i, (sample, path) in enumerate(samples.items()):
    print(f"#### {sample} ####")
    print(f"##  Converting {sample} ##")
    os.makedirs(f"{out}/{sample}", exist_ok=True)
    outpath = f"{out}/{sample}"

    # Read Scale Factor
    file_path = f"{path}/spatial/scalefactors_json.json"
    try:
        with open(file_path, 'r') as file:
            data = json.load(file)
        print(data)
    except FileNotFoundError:
        print("File not found. Please check the file path.")
    except json.JSONDecodeError:
        print("File is not a valid JSON format.")
    scale_factor = int(data["spot_diameter_fullres"])

    # Prepare Input for scResolve
    !scresolve convert visium \
    --bc-matrix "{path}/filtered_feature_bc_matrix.h5" \
    --image "{path}/spatial/tissue_lowres_image.png" \
    --tissue-positions "{path}/spatial/tissue_positions_list.csv" \
    --scale-factors "{path}/spatial/scalefactors_json.json" \
    --save-path "{out}/{sample}" \
    --mask-file "{path}/spatial/detected_tissue_image.jpg" \
    --scale {scale_factor}

Issue: The command works but eventually, the script consumes all available RAM, leading to a system crash. This happens for lowres and hires visium images.

Troubleshooting Steps Taken:

• Tried high res & low res img
• tried different values for scale, as the documentation on why a scale and scale factors parameter is needed is not very clear
• Tried optimizing the input data (e.g., subsetting tissue positions to only under tissue spots).

Questions/Requests:

1. Are there any known memory optimization techniques specific to scresolve that I can apply?
2. Is there a way to reduce the memory footprint of the convert visium command?
3. how should the scale parameter be used and how can I optimize it for memory usage
4. Any suggestions would be greatly appreciated.

Thank you for your assistance!

[DorisZhan]

I had the same issue! Did you solve it?