search

broadinstitute / CP186-A549-WG

BSD 3-Clause "New" or "Revised" License
2 stars 0 forks source link

Track recipe time & memory usage #6

Open ErinWeisbart opened 2 years ago

ErinWeisbart commented 2 years ago

Dataset has ~92 sites with data * 51 wells = ~4692 sites in 9 plates.

On a m5.8xlarge instance (128 GB. memory, 32 vCPU) all of 0.preprocess-sites completed in ~18hrs using a max of 4% memory and 3% CPU

ErinWeisbart commented 2 years ago

1.generate-profiles/0.merge-single-cells needs a lot of memory to run with output_one_single_cell_file_only: true (see #5 ). With false it runs using <2% memory of m5.8xlarge.

On m5.8xlarge (128 GB memory), 1.generate-profiles/1.aggregate hit 95% memory and crashed after 1 hr of running.

ErinWeisbart commented 2 years ago

1.generate-profiles/1.aggregate on r4.16xlarge (488 GB memory) ran for 4 hrs, made it through appending all sites, and ran out of memory making the single cell dataframe and crashed.

ErinWeisbart commented 2 years ago

1.generate-profiles/1.aggregate on x1.16xlarge (976 GB) ran for 4 hrs, used 75% memory (according to CloudWatch Agent). Like previous attempt, made it through appending all sites, and crashed making the single cell dataframe. Crash was silent which suggests it's an out-of-memory error, though that is at odds with what CloudWatch Agent shows...

ErinWeisbart commented 2 years ago

Using r4.16xlarge (488 GB memory) and per-plate basis (in config split: profile: plates: true) for 9 plates: 1.generate-profiles/1.aggregate used ~34% memory max per plate in ~8 hrs. 1.generate-profiles/2.normalize used 2% memory max per plate in ~5 hrs. (levels: gene & guide) 1.generate-profiles/3.feature-select used 2% memory max per plate in ~10 hrs.

ErinWeisbart commented 2 years ago

1.generate-profiles/0.merge-single-cells With config set to split: profile: plates: true AND output_one_single_cell_file_only: true Using r4.16xlarge (488 GB memory) Took 53 hrs, used ~25% memory.

ErinWeisbart commented 2 years ago

Using r4.16xlarge (488 GB memory) and per-plate basis (in config split: profile: plates: true) for 9 plates. 1.generate-profiles/1.normalize (levels: single_cell, output_single_cell_by_guide: true) Used ~30% memory and ~10 hrs/plate for normalize step. Used ~12% memory and ~24 hrs/plate for output_single_cell_by_guide step.