Added TRTWrapper - Githubissues

borisfom commented 3 months ago

Description

Added alternative class to ONNX->TRT export and wrap TRT engines for inference. It encapsulates filesystem persistence and does not rely on torch-tensortrt for execution. Also can be used to run ONNX with onnxruntime.

Types of changes

[x] Non-breaking change (fix or new feature that would not break existing functionality).
[ ] Breaking change (fix or new feature that would cause existing functionality to change).
[ ] New tests added to cover the changes.
[ ] Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
[ ] Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
[ ] In-line docstrings updated.
[ ] Documentation updated, tested make html command in the docs/ folder.

yiheng-wang-nv commented 3 months ago

/black

yiheng-wang-nv commented 3 months ago

Hi @borisfom , do you mind we hold this PR for some times, and after vista3d network is merged into MONAI, we can add related TRT tests here to ensure the quality.

borisfom commented 3 months ago

@yiheng-wang-nv : Sure, it's up to you when ti integrate it. Yes there should be tests - are there any networks in the core that can be used for it ?

Nic-Ma commented 3 months ago

Hi @yiheng-wang-nv ,

I see the PR still has some CI errors, could you please help review it again?

Thanks in advance.

Nic-Ma commented 3 months ago

Hi @binliunls ,

Please also help review this PR and think about how to combine with the existing API: https://github.com/Project-MONAI/MONAI/blob/dev/monai/networks/utils.py#L861

Thanks in advance.

Nic-Ma commented 3 months ago

Hi @borisfom ,

Once we have unified / enhanced new TRT API, could you please help also update the existing tutorial notebook? https://github.com/Project-MONAI/tutorials/blob/main/acceleration/TensorRT_inference_acceleration.ipynb @mingxueg-nv and @binliunls can help verify / test it.

Thanks in advance.

yiheng-wang-nv commented 3 months ago

Hi @yiheng-wang-nv ,

I see the PR still has some CI errors, could you please help review it again?

Thanks in advance.

The error seems not relate to this PR:

2024-08-09T02:36:19.2400966Z ERROR: test_gen_patches_0 (tests.test_masked_patch_wsi_dataset.TestSlidingPatchWSIDatasetCuCIM)
2024-08-09T02:36:19.2402289Z ----------------------------------------------------------------------
2024-08-09T02:36:19.2402973Z Traceback (most recent call last):
2024-08-09T02:36:19.2405048Z   File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/parameterized/parameterized.py", line 620, in standalone_func
2024-08-09T02:36:19.2406534Z     return func(*(a + p.args), **p.kwargs, **kw)
2024-08-09T02:36:19.2407783Z   File "/home/runner/work/MONAI/MONAI/tests/test_masked_patch_wsi_dataset.py", line 90, in test_gen_patches
2024-08-09T02:36:19.2408950Z     for i, sample in enumerate(dataset):
2024-08-09T02:36:19.2409969Z   File "/home/runner/work/MONAI/MONAI/monai/data/dataset.py", line 108, in __getitem__
2024-08-09T02:36:19.2410966Z     return self._transform(index)
2024-08-09T02:36:19.2578731Z   File "/home/runner/work/MONAI/MONAI/monai/data/wsi_datasets.py", line 155, in _transform
2024-08-09T02:36:19.2579862Z     image, metadata = self._get_data(sample)
2024-08-09T02:36:19.2580933Z   File "/home/runner/work/MONAI/MONAI/monai/data/wsi_datasets.py", line 148, in _get_data
2024-08-09T02:36:19.2582440Z     return self.wsi_reader.get_data(wsi=wsi_obj, location=location, size=size, level=level)
2024-08-09T02:36:19.2583789Z   File "/home/runner/work/MONAI/MONAI/monai/data/wsi_reader.py", line 385, in get_data
2024-08-09T02:36:19.2585144Z     patch = self._get_patch(each_wsi, location=location, size=size, level=level, dtype=dtype_np, mode=mode)
2024-08-09T02:36:19.2586535Z   File "/home/runner/work/MONAI/MONAI/monai/data/wsi_reader.py", line 633, in _get_patch
2024-08-09T02:36:19.2587949Z     return self.reader._get_patch(wsi=wsi, location=location, size=size, level=level, dtype=dtype, mode=mode)
2024-08-09T02:36:19.2589383Z   File "/home/runner/work/MONAI/MONAI/monai/data/wsi_reader.py", line 810, in _get_patch
2024-08-09T02:36:19.2590665Z     patch: np.ndarray = wsi.read_region(
2024-08-09T02:36:19.2591452Z ValueError: Expected int64 array-like

Hi @KumoLiu @ericspod , have you met same issues before?

KumoLiu commented 3 months ago

Hi @KumoLiu @ericspod , have you met same issues before?

Fixed in https://github.com/Project-MONAI/MONAI/pull/8007

Nic-Ma commented 2 months ago

Hi @binliunls , could you please help review the PR again?

Thanks in advance.