search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.88k stars 2.95k forks source link

[Training] Cannot generate ONNX graph for training #22188

Open srijanie03 opened 2 months ago

srijanie03 commented 2 months ago

Describe the issue

I am trying to generate a training graph by specifying the model path directly. It works correctly on CUDA (12.4) but when I run the same on Windows 11, I get an error as this:

AttributeError: 'str' object has no attribute 'graph'. When I print the forward graph I can clearly see the attribute 'graph' in it.

It does not seem to be model specific as this issue occurs for every graph I try by specifying the model path. Doing onnx.load(onnx_model_path) works but not for a model that is bigger than 2GB.

To reproduce:

https://github.com/microsoft/onnxruntime-training-examples/blob/master/on_device_training/desktop/python/mnist.ipynb

#Backward graph artifacts.generate_artifacts( "mnist.onnx", optimizer=artifacts.OptimType.AdamW, loss=artifacts.LossType.CrossEntropyLoss, requires_grad=requires_grad, frozen_params=frozen_params, artifact_directory="mnist", additional_output_names=["output"])

Urgency

Very urgent

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.19.0

PyTorch Version

2.4.0

Execution Provider

Default CPU

Execution Provider Library Version

CUDA 12.4

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.