Support for pytorch running on Apple silicon processors

[jopeptid]

Hi,

Thanks for this great tool. I was wondering if you consider supporting the Apple M1 pytorch build (currently running pytorch 1.14.0.dev20221130 and pyro-ppl 1.8.3).

The first part of the example script runs fine but then it produces an error:

AttributeError: module 'pyro.optim' has no attribute 'OneCycleLR'

Full output:

remove_background % cellbender remove-background \ --input ./tiny_raw_gene_bc_matrices/GRCh38 \ --output ./tiny_10x_pbmc.h5 \ --expected-cells 500 \ --total-droplets-included 5000 cellbender:remove-background: Command: cellbender remove-background --input ./tiny_raw_gene_bc_matrices/GRCh38 --output ./tiny_10x_pbmc.h5 --expected-cells 500 --total-droplets-included 5000 cellbender:remove-background: 2022-12-01 14:38:51 cellbender:remove-background: Running remove-background cellbender:remove-background: Loading data from directory ./tiny_raw_gene_bc_matrices/GRCh38 cellbender:remove-background: CellRanger v2 format cellbender:remove-background: Trimming dataset for inference. cellbender:remove-background: Including 100 genes that have nonzero counts. /Users/JA/CellBender/cellbender/remove_background/data/dataset.py:1436: FutureWarning: Unlike other reduction functions (e.g. skew, kurtosis), the default behavior of mode typically preserves the axis it acts along. In SciPy 1.11.0, this behavior will change: the default value of keepdims will become False, the axis over which the statistic is taken will be eliminated, and the value None will no longer be accepted. Set keepdims to True or False to avoid this warning. empty_log_counts = mode(np.round(np.log1p(counts[counts > cut]), cellbender:remove-background: Prior on counts in empty droplets is 53 cellbender:remove-background: Prior on counts for cells is 8830 cellbender:remove-background: Excluding barcodes with counts below 26 cellbender:remove-background: Using 500 probable cell barcodes, plus an additional 4500 barcodes, and 796 empty droplets. cellbender:remove-background: Largest surely-empty droplet has 36 UMI counts. Traceback (most recent call last): File "/Users/JA/anaconda3/envs/torch-gpu/bin/cellbender", line 33, in sys.exit(load_entry_point('cellbender', 'console_scripts', 'cellbender')()) File "/Users/JA/CellBender/cellbender/base_cli.py", line 101, in main cli_dict[args.tool].run(args) File "/Users/JA/CellBender/cellbender/remove_background/cli.py", line 109, in run main(args) File "/Users/JA/CellBender/cellbender/remove_background/cli.py", line 204, in main run_remove_background(args) File "/Users/JA/CellBender/cellbender/remove_background/cli.py", line 166, in run_remove_background inferred_model = run_inference(dataset_obj, args) File "/Users/JA/CellBender/cellbender/remove_background/train.py", line 294, in run_inference scheduler = pyro.optim.OneCycleLR(scheduler_args) AttributeError: module 'pyro.optim' has no attribute 'OneCycleLR'

[sjfleming]

Hi. @jopeptid , thanks for posting. I wonder if your request is related to this #149 ?

The error you posted is something I haven't seen before though. I use the OneCycle learning rate scheduler for training, and it looks like OneCycleLR is still part of pyro-ppl 1.8.3 https://docs.pyro.ai/en/1.8.3/optimization.html#pyro.optim.pytorch_optimizers.OneCycleLR so I'm not sure what that error means.

Perhaps it's happening because Pyro wraps the schedulers that pytorch has, and maybe the M1 build of pytorch does not include the one-cycle learning rate scheduler (yet...?)? I'm not sure where to find this information.

[sjfleming]

But I also think there are still other things missing from the Apple M1 build of pytorch (see #149 ) that might prevent it from working, even if this exact error were solved.

[sjfleming]

This was my request for an operation that apparently needs to be implemented still

https://github.com/pytorch/pytorch/issues/77764#issuecomment-1297310794

[jopeptid]

Thank you for the prompt response.

I don't think it is pytorch related because I can do

from torch.optim.lr_scheduler import OneCycleLR and actually use it as a scheduler on some test data without error.

But if i try

from pyro.optim import OneCycleLR Traceback (most recent call last): File "", line 1, in ImportError: cannot import name 'OneCycleLR' from 'pyro.optim' (/Users/JA/anaconda3/envs/torch-gpu/lib/python3.8/site-packages/pyro/optim/init.py)

The same code runs fine in a cellbender install on a linux box. So I would guess the pyro wrapper is looking for some other modules that may not have been ported to pytorch 1.14.0 yet?

[sjfleming]

This is a really great catch. I wonder if this might be worth making into an issue at https://github.com/pyro-ppl/pyro/issues ?

[jopeptid]

Hi Stephen,

The issue has been resolved. Pytorch was updated to to correctly wrap newer torch releases.

I had a question related to the M1 chip. Are you planning to support torch.device("mps”) in future CellBender releases?

Thank you for your support

Best, Josef

On Dec 8, 2022, at 16:01, Stephen Fleming @.***> wrote:

This is a really great catch. I wonder if this might be worth making into an issue at https://github.com/pyro-ppl/pyro/issueshttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_pyro-2Dppl_pyro_issues&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=GJwWmBLUmtBNBxAiSPPfE9zTkwSlHz2bEKQm-QzwCPDkqo7EcZvchpqyCWsmUNWY&s=WCFuNZ6ucUf1mUi5YYeG-dmaAKd-PkRxkLcQDXRcm5c&e= ?

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_broadinstitute_CellBender_issues_171-23issuecomment-2D1343346705&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=GJwWmBLUmtBNBxAiSPPfE9zTkwSlHz2bEKQm-QzwCPDkqo7EcZvchpqyCWsmUNWY&s=XrBR-J1avDBNwr8sR-Ik0t4fa3JhRGlgEmaxLfjr3ag&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIRGN2XQKKEMAHN76N76WV3WMJECZANCNFSM6AAAAAASRFLSGI&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=GJwWmBLUmtBNBxAiSPPfE9zTkwSlHz2bEKQm-QzwCPDkqo7EcZvchpqyCWsmUNWY&s=aGvpoLaWFrQkReyk09zoadici9XDg6blt2L-vLgkYpA&e=. You are receiving this because you were mentioned.Message ID: @.***>

[sjfleming]

Hi Josef, thanks for your follow-up, and I am glad to hear that particular error has been fixed.

As far as M1 support, I would love to make it happen... but it may be a little ways off. You can see the status of my own attempts here #149

Unfortunately I think we will have to wait for a lot more of the PyTorch work to be completed here: https://github.com/pytorch/pytorch/issues/77764

[ChaseTaylor939]

FYI, I was having an error today that read: "AttributeError: module 'pyro.optim' has no attribute 'OneCycleLR'".

PyTorch recently updated to version = 2.0.

I installed a previous version of PyTorch by running:

conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 cpuonly -c pytorch

based on suggestions from the PyTorch website: https://pytorch.org/get-started/previous-versions/

I think this fixed the problem. It is running now.

[jopeptid]

Great! I just tested it with the latest PyTorch release. Works flawlessly.

Thank you, Josef

On Mar 31, 2023, at 15:44, ChaseTaylor939 @.***> wrote:

FYI, I was having an error today that read: "AttributeError: module 'pyro.optim' has no attribute 'OneCycleLR'".

Pytorch recently updated to version = 2.0.

I ran:

conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 cpuonly -c pytorch

based on suggestions from the PyTorch website: https://pytorch.org/get-started/previous-versions/https://urldefense.proofpoint.com/v2/url?u=https-3A__pytorch.org_get-2Dstarted_previous-2Dversions_&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=xuq3r5IbMTujfqx9kzHsFoKinyslY7WPVSFrts0jYPy42P6AL-6TKHWKR55SwISc&s=KpbET2pIOYhenRfng9BVsRYumwavBa9LEmVYt-jyYBY&e=

I think this fixed the problem. It is running now.

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_broadinstitute_CellBender_issues_171-23issuecomment-2D1492507939&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=xuq3r5IbMTujfqx9kzHsFoKinyslY7WPVSFrts0jYPy42P6AL-6TKHWKR55SwISc&s=LlE7EzWqKDQN_WlqzYCdc-iwm_Jd7P1sXKo5hwlfcmI&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIRGN2VPLKYKCM2JDAFQBRLW64XYHANCNFSM6AAAAAASRFLSGI&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=xuq3r5IbMTujfqx9kzHsFoKinyslY7WPVSFrts0jYPy42P6AL-6TKHWKR55SwISc&s=ZdBJF5XxHYmzbrLmDLovurKw_eMRnUjfDHVqy34XJGA&e=. You are receiving this because you were mentioned.Message ID: @.***>

[sjfleming]

Hi @jopeptid , what is it that's now working with the latest pytorch release? The --mps flag on the M1 gpu from #149 ? Or were you saying that the OneCycleLR problem is solved?

[jopeptid]

Hi Stephen,

The OneCycleLR problem is gone. But it only runs on the cpu. The mps flag is not recognized.

Best, Josef

Josef Anrather BMRI Weill Cornell Medicine

---

From: Stephen Fleming @.> Sent: Wednesday, April 5, 2023 9:34:17 AM To: broadinstitute/CellBender @.> Cc: Josef Anrather @.>; Mention @.> Subject: [EXTERNAL] Re: [broadinstitute/CellBender] Support for pytorch running on Apple silicon processors (Issue #171)

Hi @jopeptidhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jopeptid&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=NMwblz41cDxP6trXjlUC7VM8Y1lcK9nt4l5yfFaQk7lAZ2M1tZrgTGcc_eh-j71A&s=INONyRHfif720r909Bq8bNnW5JRHWS0YEzUW0dK7ug8&e= , what is it that's now working with the latest pytorch release? The --mps flag on the M1 gpu from #149https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_broadinstitute_CellBender_issues_149&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=NMwblz41cDxP6trXjlUC7VM8Y1lcK9nt4l5yfFaQk7lAZ2M1tZrgTGcc_eh-j71A&s=yKA6C03R1udLXGNvEtizCdYGrnncPgwl_lVggAdceGM&e= ? Or were you saying that the OneCycleLR problem is solved?

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_broadinstitute_CellBender_issues_171-23issuecomment-2D1497495862&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=NMwblz41cDxP6trXjlUC7VM8Y1lcK9nt4l5yfFaQk7lAZ2M1tZrgTGcc_eh-j71A&s=f4MeIiUFgXvwIES_x86zYEHcF7zkUAeDc3WNOvHBGVA&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIRGN2VY7Z3FNFIS4DBSVJDW7VYFTANCNFSM6AAAAAASRFLSGI&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=NMwblz41cDxP6trXjlUC7VM8Y1lcK9nt4l5yfFaQk7lAZ2M1tZrgTGcc_eh-j71A&s=wQiUsNAjec92oFbNs1wqivl4dBBVFtylBuQprtWU5i8&e=. You are receiving this because you were mentioned.Message ID: @.***>

[jopeptid]

To be precise.

this isn’t completing

cellbender remove-background \ --input ./tiny_raw_gene_bc_matrices/GRCh38 \ --output ./tiny_10x_pbmc.h5 \ --expected-cells 500 \ --mps \ --total-droplets-included 5000

It produces the following error

cellbender: error: unrecognized arguments: --mps

BUT this runs fine on the m1 gpu using the same environment

!/usr/bin/env python3

-- coding: utf-8 --

import torch import math

this ensures that the current MacOS version is at least 12.3+

print(torch.backends.mps.is_available())

dtype = torch.float device = torch.device("mps")

Create random input and output data

x = torch.linspace(-math.pi, math.pi, 2000, device=device, dtype=dtype) y = torch.sin(x)

Create random input and output data

x = torch.linspace(-math.pi, math.pi, 2000, device=device, dtype=dtype) y = torch.sin(x)

Randomly initialize weights

a = torch.randn((), device=device, dtype=dtype) b = torch.randn((), device=device, dtype=dtype) c = torch.randn((), device=device, dtype=dtype) d = torch.randn((), device=device, dtype=dtype)

learning_rate = 1e-6

for t in range(2000):

Forward pass: compute predicted y

            y_pred = a + b * x + c * x ** 2 + d * x ** 3

            # Compute and print loss
            loss = (y_pred - y).pow(2).sum().item()

            # Backprop to compute gradients of a, b, c, d with respect to loss
            grad_y_pred = 2.0 * (y_pred - y)
            grad_a = grad_y_pred.sum()
            grad_b = (grad_y_pred * x).sum()
            grad_c = (grad_y_pred * x ** 2).sum()
            grad_d = (grad_y_pred * x ** 3).sum()

            # Update weights using gradient descent
            a -= learning_rate * grad_a
            b -= learning_rate * grad_b
            c -= learning_rate * grad_c
            d -= learning_rate * grad_d

print(f'Result: y = {a.item()} + {b.item()} x + {c.item()} x^2 + {d.item()} x^3')

From: Josef Anrather @.> Date: Wednesday, April 5, 2023 at 09:45 To: broadinstitute/CellBender @.>, broadinstitute/CellBender @.> Cc: Mention @.> Subject: Re: [EXTERNAL] Re: [broadinstitute/CellBender] Support for pytorch running on Apple silicon processors (Issue #171) Hi Stephen,

The OneCycleLR problem is gone. But it only runs on the cpu. The mps flag is not recognized.

Best, Josef

Josef Anrather BMRI Weill Cornell Medicine

---

From: Stephen Fleming @.> Sent: Wednesday, April 5, 2023 9:34:17 AM To: broadinstitute/CellBender @.> Cc: Josef Anrather @.>; Mention @.> Subject: [EXTERNAL] Re: [broadinstitute/CellBender] Support for pytorch running on Apple silicon processors (Issue #171)

Hi @jopeptidhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jopeptid&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=NMwblz41cDxP6trXjlUC7VM8Y1lcK9nt4l5yfFaQk7lAZ2M1tZrgTGcc_eh-j71A&s=INONyRHfif720r909Bq8bNnW5JRHWS0YEzUW0dK7ug8&e= , what is it that's now working with the latest pytorch release? The --mps flag on the M1 gpu from #149https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_broadinstitute_CellBender_issues_149&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=NMwblz41cDxP6trXjlUC7VM8Y1lcK9nt4l5yfFaQk7lAZ2M1tZrgTGcc_eh-j71A&s=yKA6C03R1udLXGNvEtizCdYGrnncPgwl_lVggAdceGM&e= ? Or were you saying that the OneCycleLR problem is solved?

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_broadinstitute_CellBender_issues_171-23issuecomment-2D1497495862&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=NMwblz41cDxP6trXjlUC7VM8Y1lcK9nt4l5yfFaQk7lAZ2M1tZrgTGcc_eh-j71A&s=f4MeIiUFgXvwIES_x86zYEHcF7zkUAeDc3WNOvHBGVA&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIRGN2VY7Z3FNFIS4DBSVJDW7VYFTANCNFSM6AAAAAASRFLSGI&d=DwMCaQ&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=V_0MVBYdM5AEM9BiKHL_8A0King1YUuaFVX2OkU23gs&m=NMwblz41cDxP6trXjlUC7VM8Y1lcK9nt4l5yfFaQk7lAZ2M1tZrgTGcc_eh-j71A&s=wQiUsNAjec92oFbNs1wqivl4dBBVFtylBuQprtWU5i8&e=. You are receiving this because you were mentioned.Message ID: @.***>

[sjfleming]

Okay makes sense. If you try the branch sf_pytorch_mps_backend from this repository, you'll see that the --mps input argument should exist. But it will (probably) still not actually work. Here (https://github.com/broadinstitute/CellBender/issues/149#issuecomment-1297271227) @asabjorklund saw some errors, stemming from the fact that there are some operations that are still not implemented for the M1 backend.

[Nahviss]

Hi all - I'm also new to cellbender and python (indeed, to single-cell work and all of statistical coding). Hopefully this question is topical here... Not foreseeing cellbender in my future, I wound up grabbing a ryzen 5 amd machine for personal use/off hour data crunching with no thought to graphics output/the GPU... what I am seeing online suggests GPU computing with amd systems is possible (directML?); is that more or less true and is there any hope cellbender might become compatible with amd GPUs??

Thanks for any education!

SF Washington University, St. Louis

[sjfleming]

Hi @Nahviss , I am no expert on this topic myself, as I have only used Nvidia GPUs in the past.

I see some information here which might be relevant: https://discuss.pytorch.org/t/how-to-run-torch-with-amd-gpu/157069/3 and here: https://github.com/pytorch/pytorch/issues/10670

The key thing needed to run cellbender is a working installation of pytorch. (And the good news is that there is a massive user community for pytorch. I'm betting if it's possible, someone will have a post about how to do it...) If you come up with a way to install pytorch on your machine so that the command

import torch
torch.cuda.is_available()

returns True, then you'll be good-to-go in terms of running cellbender (and any other pytorch code) on your AMD gpu.

[Nahviss]

@sjfleming thanks much for the tip. Will post here if I manage to get it working!

Update: To no one's surprise (most likely) I did not get the AMD GPU computing working, and I didn't try too hard either. I did pick up a 4080 laptop instead.