WayScience / CytoSnake

Orchestrating high-dimensional cell morphology data processing pipelines
https://cytosnake.readthedocs.io
Creative Commons Attribution 4.0 International
3 stars 3 forks source link

Allow CytoSnake to default to `mamba` as default package manager #45

Closed axiomcura closed 1 year ago

axiomcura commented 1 year ago

This PR introduces a new feature that allows CytoSnake to create environments using the mamba environment manager.

Motivation

As mentioned in #34, conda has a tendency to get extremely slow to create environments as the number of dependencies increase.

Usage

Switching to mamba is very simple! Users need to go to the configs/configuration.yaml file and look for the env_manager keyword.

config_name: cytosnake_configs

env_manager: conda

# trucated for demonstrations purposes

Then edit the env_manager to mamba in the configs:

config_name: cytosnake_configs

env_manager: mamba

# trucated for demonstrations purposes

And that's it!

benchmarking

Here are some tests that show the complete runtime of a workflow.

The Linux time command was used to quantify runtime.

Below are the results:

Testing with cp_process_singlecells using NF1 data (SQLITE -> PARQUET)

conda:

real    3m7.812s
user    2m22.196s
sys     0m18.924s

mamba

real    1m40.792s
user    1m6.971s
sys     0m14.046s

Even though the time difference is quite small, adding more software dependencies will cause conda to be exponentially longer than mamba when creating environments.

additional notes

Snakemake provides an output indicating whether mamba is being used. here's an example below:

Activating conda environment: mamba/f9037c5b00e972a33df92e433d107ff8_
WARNING:snakemake.logging:Activating conda environment: mamba/f9037c5b00e972a33df92e433d107ff8_
Activating conda environment: mamba/f9037c5b00e972a33df92e433d107ff8_
WARNING:snakemake.logging:Activating conda environment: mamba/f9037c5b00e972a33df92e433d107ff8_
axiomcura commented 1 year ago

@gwaybio hopefully I have answered all your comments and suggestions. Let me know if there are any changes required.

If not, I'll merge this tomorrow, morning.

gwaybio commented 1 year ago

I have answered all your comments and suggestions. Let me know if there are any changes required.

I responded to two quick followups - up to you when to merge. You already have my approval!