search

andresprados / SPIGA

SPIGA: Shape Preserving Facial Landmarks with Graph Attention Networks.
https://bmvc2022.mpi-inf.mpg.de/155/
BSD 3-Clause "New" or "Revised" License
304 stars 32 forks source link

ONNX Model Availability #3

Closed john09282922 closed 1 year ago

andresprados commented 1 year ago

Hi @john09282922,

Not yet, I believe that it shouldn't be a complex conversion since most of the layers are common ones. Maybe the visual feature cropping used as GNN input could not be supported but is the same method used by well knonw models like MaskRCNN.

Depending on the impact of the repository, we will consider further improvements of it surch as releasing ONNX models. In the meantime, feel free to submit a PR if you do the conversion.

Best, Andrés

john09282922 commented 1 year ago

Hi, thank you for appreciating your reply. Would you mind telling me what kind of part I have to replace your gnn input with MaskRcnn? I need your help... plz let me know

thanks, jungmin

On Tue, Mar 14, 2023 at 12:47 PM Andrés Prados Torreblanca < @.***> wrote:

Hi @john09282922 https://github.com/john09282922,

Not yet, I believe that it shouldn't be a complex conversion since most of the layers are common ones. Maybe the visual feature cropping used as GNN input could not be supported but is the same method used by well knonw models like MaskRCNN.

Depending on the impact of the repository, we will consider further improvements of it surch as releasing ONNX models. In the meantime, feel free to submit a PR if you do the conversion.

Best, Andrés

— Reply to this email directly, view it on GitHub https://github.com/andresprados/SPIGA/issues/3#issuecomment-1468454904, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3KJF5IUNDEQKNNEKIV34DDW4COJ5ANCNFSM6AAAAAAVVMQEJE . You are receiving this because you were mentioned.Message ID: @.***>

andresprados commented 1 year ago

Sry Jungmin, it is just a guess of a possible problem since it is a sort of "dynamic operation" and ONNX usually doesnt like them. I havent try to convert the model so I cannot know if it is even a problem.

Let me know which exact error are you facing. Maybe I can help.

john09282922 commented 1 year ago

Hi, Thank you for reanswering my question.

UnsupportedOperatorError: Exporting the operator 'aten::affine_grid_generator' to ONNX opset version 14 is not supported.

This 'aten' makes error to export onnx file from your model. Is there any solution? is there put inputs all together?

thanks, jungmin

On Tue, Mar 14, 2023 at 1:48 PM Andrés Prados Torreblanca < @.***> wrote:

Reopened #3 https://github.com/andresprados/SPIGA/issues/3.

— Reply to this email directly, view it on GitHub https://github.com/andresprados/SPIGA/issues/3#event-8746854971, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3KJF5KZKWHZBQ2QAUZQAEDW4CVPHANCNFSM6AAAAAAVVMQEJE . You are receiving this because you were mentioned.Message ID: @.***>

andresprados commented 1 year ago

Jackpot! Your are facing a conversion problem related to the affine transformation used to crop the visual features of the GNN input. This operation is used by Spatial Transformer Networks, take a look at this issue, it seems to be already solved.

john09282922 commented 1 year ago

Hi, I read your issue part. unfortunately, It is problem like aten::affine_grid_generator' to ONNX opset version 16

On Thu, Mar 16, 2023 at 7:55 AM Andrés Prados Torreblanca < @.***> wrote:

Jackpot! Your are facing a conversion problem related to the affine transformation used to crop the visual features of the GNN input. This operation is used by Spatial Transformer Networks, take a look at this issue https://github.com/pytorch/pytorch/issues/27212, it seems to be already solved.

— Reply to this email directly, view it on GitHub https://github.com/andresprados/SPIGA/issues/3#issuecomment-1471818670, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3KJF5JAUTZVTP33A6WJTTLW4L5UHANCNFSM6AAAAAAVVMQEJE . You are receiving this because you were mentioned.Message ID: @.***>

john09282922 commented 1 year ago

Hi, Unfortunately, this is not working following the issue. the issue handled on grid sample, not grid generator. instead of using grid? is there other way? I changed to onnx version 16, even 17... it is not works ....

thanks, jungmin

On Thu, Mar 16, 2023 at 7:55 AM Andrés Prados Torreblanca < @.***> wrote:

Jackpot! Your are facing a conversion problem related to the affine transformation used to crop the visual features of the GNN input. This operation is used by Spatial Transformer Networks, take a look at this issue https://github.com/pytorch/pytorch/issues/27212, it seems to be already solved.

— Reply to this email directly, view it on GitHub https://github.com/andresprados/SPIGA/issues/3#issuecomment-1471818670, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3KJF5JAUTZVTP33A6WJTTLW4L5UHANCNFSM6AAAAAAVVMQEJE . You are receiving this because you were mentioned.Message ID: @.***>

john09282922 commented 1 year ago

or am i making some problem to make random input?

thanks, jungmin

sample_input1 = torch.rand((1, 3, 256, 256)) sample_input2 = torch.rand((1, 98, 3)) sample_input3 = torch.rand((1, 3, 3))

sample_input = [sample_input1, sample_input2, sample_input3]

On Thu, Mar 16, 2023 at 3:30 PM JUNGMIN HWANG @.***> wrote:

Hi, Unfortunately, this is not working following the issue. the issue handled on grid sample, not grid generator. instead of using grid? is there other way? I changed to onnx version 16, even 17... it is not works ....

thanks, jungmin

On Thu, Mar 16, 2023 at 7:55 AM Andrés Prados Torreblanca < @.***> wrote:

Jackpot! Your are facing a conversion problem related to the affine transformation used to crop the visual features of the GNN input. This operation is used by Spatial Transformer Networks, take a look at this issue https://github.com/pytorch/pytorch/issues/27212, it seems to be already solved.

— Reply to this email directly, view it on GitHub https://github.com/andresprados/SPIGA/issues/3#issuecomment-1471818670, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3KJF5JAUTZVTP33A6WJTTLW4L5UHANCNFSM6AAAAAAVVMQEJE . You are receiving this because you were mentioned.Message ID: @.***>

edmosRovi commented 1 year ago

Hello,

I am running into the same problem, and I used the affine_grad reimplementation from https://github.com/pytorch/pytorch/issues/30563.

I was able to convert the model to ONNX using opset 16, but when I used it with onnxruntime, the session creation failed: "Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from model.onnx failed:Node (/Squeeze) Op (Squeeze) [ShapeInferenceError] Dimension of input 1 must be 1 instead of 2".

Any idea how I can work with the model in onnxruntime? My ultimate goal is to generate a tensorrt engine from the model, but I am doing it step by step first.

Thanks.

john09282922 commented 1 year ago

Hello,

I am running into the same problem, and I used the affine_grad reimplementation from pytorch/pytorch#30563.

I was able to convert the model to ONNX using opset 16, but when I used it with onnxruntime, the session creation failed: "Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from model.onnx failed:Node (/Squeeze) Op (Squeeze) [ShapeInferenceError] Dimension of input 1 must be 1 instead of 2".

Any idea how I can work with the model in onnxruntime? My ultimate goal is to generate a tensorrt engine from the model, but I am doing it step by step first.

Thanks.

Hi, could you share how to change affine grid reimplementation? and did you set up input correctly?

edmosRovi commented 1 year ago

Hi,

To change affine grid, I did it in "spiga\models\spiga.py", in "extract_visual_embedded", replaces "grid = torch.nn.functional.affine_grid(theta, (B * L, C, self.kwindow, self.kwindow))" with the call to the reimplemented method.

To set up the input, I did this:

model_cfg = ModelConfig('merlrav') loader_3DM = AddModel3D(model_cfg.dataset.ldm_ids, ftmap_size=model_cfg.ftmap_size, focal_ratio=model_cfg.focal_ratio, totensor=True) params_3DM = _data2device(loader_3DM()) model3d = params_3DM['model3d'] cam_matrix = params_3DM['cam_matrix'] image = cv2.imread('img.jpg') bboxes = [[274.84974931074015, 128.89023861421734, 203.57020487055968, 295.73514067406427]] inputs, crop_bboxes = pretreat(image, bboxes)

Unfortunately, I don't think this is a proper way to do it, as I can't use the onnx model in any way. Neither with OpenCV DNN, nor with ONNXRuntime, nor TensorRT.

Any idea how it could be done?

john09282922 commented 1 year ago

Hi,

To change affine grid, I did it in "spiga\models\spiga.py", in "extract_visual_embedded", replaces "grid = torch.nn.functional.affine_grid(theta, (B * L, C, self.kwindow, self.kwindow))" with the call to the reimplemented method.

To set up the input, I did this:

model_cfg = ModelConfig('merlrav') loader_3DM = AddModel3D(model_cfg.dataset.ldm_ids, ftmap_size=model_cfg.ftmap_size, focal_ratio=model_cfg.focal_ratio, totensor=True) params_3DM = _data2device(loader_3DM()) model3d = params_3DM['model3d'] cam_matrix = params_3DM['cam_matrix'] image = cv2.imread('img.jpg') bboxes = [[274.84974931074015, 128.89023861421734, 203.57020487055968, 295.73514067406427]] inputs, crop_bboxes = pretreat(image, bboxes)

Unfortunately, I don't think this is a proper way to do it, as I can't use the onnx model in any way. Neither with OpenCV DNN, nor with ONNXRuntime, nor TensorRT.

Any idea how it could be done?

did u check that original code's input and your input size is same? anyway, which lines did you change to export spiga model to onnx ?

SaShukla090 commented 1 year ago

Did anybody able to generate the ONNX model of SPIGA, which successfully runs on ONNX runtime or TensorRT?. Please reply and let us know your changes to convert the model to ONNX type. Any help will be appreciated!

lokeycookie commented 1 year ago

Someone posted to use dynamo_export here. However, i got the following error:

SPIGA model loaded! /home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/nn/functional.py:4358: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details. warnings.warn( /home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/nn/functional.py:4296: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details. warnings.warn( EXPORTING!!!!!!! /home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py:130: UserWarning: torch.onnx.dynamo_export only implements opset version 18 for now. If you need to use a different opset version, please register them with register_custom_op. warnings.warn( /home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/nn/functional.py:4358: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details. warnings.warn( /home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/nn/functional.py:4296: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details. warnings.warn( Traceback (most recent call last): File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1413, in run_node return getattr(args[0], node.target)(*args[1:], kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper return fn(*args, *kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1299, in __torch_dispatch__ return self.dispatch(func, types, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1504, in dispatch return decomposition_table[func](args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_refs/init.py", line 4445, in view return _reshape_view_helper(a, shape, allow_copy=False) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_refs/init.py", line 3578, in _reshape_view_helper shape = utils.infer_size(shape, a.numel()) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_prims_common/init.py", line 813, in infer_size numel == newsize or (dim is not None and newsize > 0 and numel % newsize == 0), File "/home/evas/anaconda3/envs/CHtest/lib/python3.10/site-packages/torch/init.py", line 352, in bool return self.node.bool() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolicshapes.py", line 1003, in bool return self.guard_bool("", 0) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 985, in guard_bool r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3512, in evaluate_expr concrete_val = self.size_hint(orig_expr) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3313, in size_hint raise self._make_data_dependent_error(result_expr, expr) torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.) The expression we were trying to evaluate is Eq(2i0, 98) (unhinted: Eq(2*i0, 98)). Scroll up to see where each of these data-dependent accesses originally occurred.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1350, in get_fake_value return wrap_fake_exception( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 925, in wrap_fake_exception return fn() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1351, in lambda: run_node(tx.output, node, args, kwargs, nnmodule) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1430, in run_node raise RuntimeError(fn_str + str(e)).with_traceback(e.traceback) from e File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1413, in run_node return getattr(args[0], node.target)(*args[1:], kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper return fn(*args, *kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1299, in __torch_dispatch__ return self.dispatch(func, types, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1504, in dispatch return decomposition_table[func](args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_refs/init.py", line 4445, in view return _reshape_view_helper(a, shape, allow_copy=False) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_refs/init.py", line 3578, in _reshape_view_helper shape = utils.infer_size(shape, a.numel()) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_prims_common/init.py", line 813, in infer_size numel == newsize or (dim is not None and newsize > 0 and numel % newsize == 0), File "/home/evas/anaconda3/envs/CHtest/lib/python3.10/site-packages/torch/init.py", line 352, in bool return self.node.bool() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolicshapes.py", line 1003, in bool return self.guard_bool("", 0) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 985, in guard_bool r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3512, in evaluate_expr concrete_val = self.size_hint(orig_expr) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3313, in size_hint raise self._make_data_dependent_error(result_expr, expr) RuntimeError: Failed running call_method reshape((FakeTensor(..., device='cuda:0', size=(1, i0, 2), grad_fn=), 1, 98, -1), *{}): It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.) The expression we were trying to evaluate is Eq(2i0, 98) (unhinted: Eq(2*i0, 98)). Scroll up to see where each of these data-dependent accesses originally occurred.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py", line 1195, in dynamo_export ).export() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py", line 941, in export graph_module = self.options.fx_tracer.generate_fx( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/fx/dynamo_graph_extractor.py", line 199, in generate_fx graph_module, graph_guard = torch._dynamo.export( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1150, in inner result_traced = opt_f(*args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn return fn(*args, *kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/fx/dynamo_graph_extractor.py", line 154, in wrapped return output_adapter.apply(model_func(args, kwargs)) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors return callback(frame, cache_entry, hooks, frame_state) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn return fn(*args, *kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 380, in _convert_frame_assert return _compile( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 561, in _compile guarded_code = compile_inner(code, one_graph, hooks, transform) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 197, in time_wrapper r = func(args, **kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 483, in compile_inner out_code = transform_code_object(code, transform) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object transformations(instructions, code_options) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 449, in transform tracer.run() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2093, in run super().run() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 733, in run and self.step() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 696, in step getattr(self, inst.opname)(inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 397, in wrapper return inner_fn(self, inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1124, in CALL_FUNCTION self.call_function(fn, args, {}) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 570, in call_function self.push(fn.call_function(self, args, kwargs)) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function return super().call_function(tx, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function return super().call_function(tx, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function return tx.inline_user_function_return( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 606, in inline_user_function_return result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2210, in inline_call return cls.inlinecall(parent, func, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2326, in inlinecall tracer.run() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 733, in run and self.step() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 696, in step getattr(self, inst.opname)(inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 397, in wrapper return inner_fn(self, inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1124, in CALL_FUNCTION self.call_function(fn, args, {}) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 570, in call_function self.push(fn.call_function(self, args, kwargs)) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function return super().call_function(tx, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function return super().call_function(tx, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function return tx.inline_user_function_return( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 606, in inline_user_function_return result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2210, in inline_call return cls.inlinecall(parent, func, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2326, in inlinecall tracer.run() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 733, in run and self.step() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 696, in step getattr(self, inst.opname)(inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 397, in wrapper return inner_fn(self, inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1124, in CALL_FUNCTION self.call_function(fn, args, {}) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 570, in call_function self.push(fn.call_function(self, args, kwargs)) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 594, in call_function return self.obj.call_method(tx, self.name, args, kwargs).add_options(self) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 652, in call_method return wrap_fx_proxy( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1198, in wrap_fx_proxy return wrap_fx_proxy_cls( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1285, in wrap_fx_proxy_cls example_value = get_fake_value(proxy.node, tx) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1379, in get_fake_value raise UserError( torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time.You will need to explicitly give hint to the compiler. Please take a look at constrain_as_value OR constrain_as_size APIs

from user code: File "/home/evas/Downloads/SPIGA/SPIGA/spiga/models/spiga.py", line 82, in forward embedded_ft = self.extract_embedded(pts_proj, visual_field, step) File "/home/evas/Downloads/SPIGA/SPIGA/spiga/models/spiga.py", line 124, in extract_embedded shape_ft = self.calculate_distances(pts_proj) File "/home/evas/Downloads/SPIGA/SPIGA/spiga/models/spiga.py", line 163, in calculate_distances dist_wo_self = dist[:, self.diagonal_mask, :].reshape(B, L, -1)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/evas/Downloads/SPIGA/SPIGA/app_CH.py", line 73, in tracked_obj = processor.process_frame(frame, [faces[0]]) File "/home/evas/Downloads/SPIGA/SPIGA/spiga/demo/analyze/extract/spiga_processor.py", line 35, in process_frame features = self.processor.inference(frame, bboxes) File "/home/evas/Downloads/SPIGA/SPIGA/spiga/inference/framework.py", line 68, in inference outputs = self.net_forward(batch_crops) File "/home/evas/Downloads/SPIGA/SPIGA/spiga/inference/framework.py", line 97, in net_forward torch.onnx.dynamo_export(self.model, inputs).save("wflw.onnx") File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py", line 1206, in dynamo_export raise OnnxExporterError(

what i add to inference/framework.py

def net_forward(self, inputs):
        outputs = self.model(inputs)
        print("EXPORTING!!!!!!!")
        export_options = torch.onnx.ExportOptions(dynamic_shapes=True)
        torch.onnx.dynamo_export(self.model, inputs).save("wflw.onnx")
        return outputs
zhenyi1998 commented 1 year ago

Someone posted to use dynamo_export here. However, i got the following error:

SPIGA model loaded! /home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/nn/functional.py:4358: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details. warnings.warn( /home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/nn/functional.py:4296: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details. warnings.warn( EXPORTING!!!!!!! /home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py:130: UserWarning: torch.onnx.dynamo_export only implements opset version 18 for now. If you need to use a different opset version, please register them with register_custom_op. warnings.warn( /home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/nn/functional.py:4358: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details. warnings.warn( /home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/nn/functional.py:4296: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details. warnings.warn( Traceback (most recent call last): File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1413, in run_node return getattr(args[0], node.target)(*args[1:], kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper return fn(args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1299, in torch_dispatch return self.dispatch(func, types, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1504, in dispatch return decomposition_table[func](_args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_refs/init.py", line 4445, in view return _reshape_view_helper(a, _shape, allow_copy=False) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_refs/init.py", line 3578, in _reshape_view_helper shape = utils.infer_size(shape, a.numel()) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_prims_common/init.py", line 813, in infer_size numel == newsize or (dim is not None and newsize > 0 and numel % newsize == 0), File "/home/evas/anaconda3/envs/CHtest/lib/python3.10/site-packages/torch/init.py", line 352, in bool return self.node.bool() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolicshapes.py", line 1003, in bool return self.guard_bool("", 0) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 985, in guard_bool r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3512, in evaluate_expr concrete_val = self.size_hint(orig_expr) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3313, in size_hint raise self._make_data_dependent_error(result_expr, expr) torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.) The expression we were trying to evaluate is Eq(2_i0, 98) (unhinted: Eq(2_i0, 98)). Scroll up to see where each of these data-dependent accesses originally occurred. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1350, in get_fake_value return wrap_fake_exception( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 925, in wrap_fake_exception return fn() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1351, in lambda: run_node(tx.output, node, args, kwargs, nnmodule) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1430, in run_node raise RuntimeError(fn_str + str(e)).with_traceback(e.traceback) from e File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1413, in run_node return getattr(args[0], node.target)(args[1:], kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper return fn(_args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1299, in torch_dispatch return self.dispatch(func, types, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1504, in dispatch return decomposition_table[func](_args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_refs/init.py", line 4445, in view return _reshape_view_helper(a, _shape, allow_copy=False) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_refs/init.py", line 3578, in _reshape_view_helper shape = utils.infer_size(shape, a.numel()) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_prims_common/init.py", line 813, in infer_size numel == newsize or (dim is not None and newsize > 0 and numel % newsize == 0), File "/home/evas/anaconda3/envs/CHtest/lib/python3.10/site-packages/torch/init.py", line 352, in bool return self.node.bool() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolicshapes.py", line 1003, in bool return self.guard_bool("", 0) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 985, in guard_bool r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3512, in evaluate_expr concrete_val = self.size_hint(orig_expr) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3313, in size_hint raise self._make_data_dependent_error(result_expr, expr) RuntimeError: Failed running callmethod reshape((FakeTensor(..., device='cuda:0', size=(1, i0, 2), grad_fn=), 1, 98, -1), {}): It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.) The expression we were trying to evaluate is Eq(2i0, 98) (unhinted: Eq(2i0, 98)). Scroll up to see where each of these data-dependent accesses originally occurred.__ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py", line 1195, in dynamo_export ).export() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py", line 941, in export graph_module = self.options.fx_tracer.generate_fx( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/fx/dynamo_graph_extractor.py", line 199, in generate_fx graph_module, graph_guard = torch._dynamo.export( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1150, in inner result_traced = opt_f(*args, *kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn return fn(args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/fx/dynamo_graph_extractor.py", line 154, in wrapped return output_adapter.apply(model_func(*args, kwargs)) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors return callback(frame, cache_entry, hooks, frame_state) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn return fn(*args, *kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 380, in _convert_frame_assert return _compile( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 561, in _compile guarded_code = compile_inner(code, one_graph, hooks, transform) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 197, in time_wrapper r = func(args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 483, in compile_inner out_code = transform_code_object(code, transform) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object transformations(instructions, code_options) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 449, in transform tracer.run() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2093, in run super().run() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 733, in run and self.step() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 696, in step getattr(self, inst.opname)(inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 397, in wrapper return inner_fn(self, inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1124, in CALL_FUNCTION self.call_function(fn, args, {}) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 570, in call_function self.push(fn.call_function(self, args, kwargs)) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function return super().call_function(tx, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function return super().call_function(tx, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function return tx.inline_user_function_return( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 606, in inline_user_function_return result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2210, in inline_call return cls.inlinecall(parent, func, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2326, in inlinecall tracer.run() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 733, in run and self.step() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 696, in step getattr(self, inst.opname)(inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 397, in wrapper return inner_fn(self, inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1124, in CALL_FUNCTION self.call_function(fn, args, {}) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 570, in call_function self.push(fn.call_function(self, args, kwargs)) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function return super().call_function(tx, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function return super().call_function(tx, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function return tx.inline_user_function_return( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 606, in inline_user_function_return result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2210, in inline_call return cls.inlinecall(parent, func, args, kwargs) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2326, in inlinecall tracer.run() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 733, in run and self.step() File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 696, in step getattr(self, inst.opname)(inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 397, in wrapper return inner_fn(self, inst) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1124, in CALL_FUNCTION self.call_function(fn, args, {}) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 570, in call_function self.push(fn.call_function(self, args, kwargs)) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 594, in call_function return self.obj.call_method(tx, self.name, args, kwargs).add_options(self) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 652, in call_method return wrap_fx_proxy( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1198, in wrap_fx_proxy return wrap_fx_proxy_cls( File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1285, in wrap_fx_proxy_cls example_value = get_fake_value(proxy.node, tx) File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1379, in get_fake_value raise UserError( torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time.You will need to explicitly give hint to the compiler. Please take a look at constrain_as_value OR constrain_as_size APIs from user code: File "/home/evas/Downloads/SPIGA/SPIGA/spiga/models/spiga.py", line 82, in forward embedded_ft = self.extract_embedded(pts_proj, visual_field, step) File "/home/evas/Downloads/SPIGA/SPIGA/spiga/models/spiga.py", line 124, in extract_embedded shape_ft = self.calculate_distances(pts_proj) File "/home/evas/Downloads/SPIGA/SPIGA/spiga/models/spiga.py", line 163, in calculate_distances dist_wo_self = dist[:, self.diagonal_mask, :].reshape(B, L, -1) Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/evas/Downloads/SPIGA/SPIGA/app_CH.py", line 73, in tracked_obj = processor.process_frame(frame, [faces[0]]) File "/home/evas/Downloads/SPIGA/SPIGA/spiga/demo/analyze/extract/spiga_processor.py", line 35, in process_frame features = self.processor.inference(frame, bboxes) File "/home/evas/Downloads/SPIGA/SPIGA/spiga/inference/framework.py", line 68, in inference outputs = self.net_forward(batch_crops) File "/home/evas/Downloads/SPIGA/SPIGA/spiga/inference/framework.py", line 97, in net_forward torch.onnx.dynamo_export(self.model, inputs).save("wflw.onnx") File "/home/evas/anaconda3/envs/CH_test/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py", line 1206, in dynamo_export raise OnnxExporterError(

what i add to inference/framework.py

def net_forward(self, inputs):
        outputs = self.model(inputs)
        print("EXPORTING!!!!!!!")
        export_options = torch.onnx.ExportOptions(dynamic_shapes=True)
        torch.onnx.dynamo_export(self.model, inputs).save("wflw.onnx")
        return outputs

Have u managed to solve this issue?