[QST] GDS, dlpack and cuCIM

[benjha]

Hi cuCIM team,

I am deploying RAPIDS with cuCIM v21.06 on ppc64le architecture (Summit cluster) and in our NV-ARM Developer Kit cluster, and noted during compilation, it grabs GDS. So my questions are:

Is GDS a hard dependency ?, if not, is there any env. variables to disable it ?

Thanks,

[gigony]

Thanks @benjha for your interest in cuCIM!

Is GDS a hard dependency ?, if not, is there any env. variables to disable it ?

GDS is not a hard dependency. GDS is already disabled for now and it would work in the compatible mode (has the same behavior but without libcufile (GDS)).

https://github.com/rapidsai/cucim/blob/45cf987e17e22acd84165e3dd3d6bd5ae72e19cd/CMakeLists.txt#L106

The current implementation uses a wrapper code for cufile and it load libcufile.so dynamically if libcufile is available (and libcufile.so won't be accessed if CUCIM_SUPPORT_GDS is OFF). https://github.com/rapidsai/cucim/blob/048d53e6f99c2129b9febd08e0ae6d1b37d74451/gds/src/cufile_stub.cpp#L55-L59

In current implementation, even if GDS (libcufile.so) is not available/needed, GDS's header file (cufile.h) is needed: https://github.com/rapidsai/cucim/blob/048d53e6f99c2129b9febd08e0ae6d1b37d74451/gds/include/cufile_stub.h#L19

To get cufile.h (for both with GDS and without GDS), a GDS package (v0.95 which is old now) is downloaded from the internet and used during building (https://github.com/gigony/cucim/blob/python_unittests/CONTRIBUTING.md#local-development-using-conda-environment-for-gcc-9x-and-nvcc-110x): https://github.com/rapidsai/cucim/blob/45cf987e17e22acd84165e3dd3d6bd5ae72e19cd/run#L392

For this reason, I think it wouldn't be a problem though you will see GDS-related messages during the build.

cuCIM currently doesn't support non-X86-64 architecture(and non-Linux OS) and has not tested on such architecture/OS. Please let us know what kinds of errors do you see during the build.

We are also interested in your use cases with cuCIM in your project and wonder if which part (image loader or image processing) would be your focus area. Thanks.

Note: Image processing part (scikit-image compatible APIs using CuPy) is pure Python code and no platform-dependent as long as CuPy is installed. We may want to provide an option to build only one of them (cucim.skimage and cucim.clara).

[benjha]

Thanks for the clarification. I agree, if we disable access to libcufile.so we'll be in the safe place.

I am supporting Rapids on Summit and I'd like to include cuCIM as part of the deployment. We have some users currently working with scikit-image and cuCIM will be a good addition so they can move their workflows to GPUs. Of course in the context of a HPC system while I/O is important for now we are focusing on image processing. I am also evaluating Rapids performance on ARM architecture, just to track performance is comparable to x86/ppc64le.

We also are evaluating image processing libraries in the context of Whole slide imaging in pathology. This will also include training/inference of tumor sites in a subsequent phase.

[benjha]

Btw,

Is cuCIM not meant to be build within an existent built from source Rapids environment ?

I noted cuCIM gets its own rmm and also uses its own dlpack (v0.2) which is different to the Rapids 21.06's supported version (dlpack>=0.5,<0.6.0a0), thus I am having this compilation error given that DLContext does not exist in dlpack 0.5

/autofs/nccs-svm1_sw/summit/ums/gen119/nvrapids_21.06_src/cucim_v21.06.00/cpp/plugins/cucim.kit.cuslide/src/cuslide/tiff/ifd.cpp:262:25: error: 'struct DLTensor' has no member named 'ctx'
  262 |     out_image_container.ctx = DLContext{ static_cast<DLDeviceType>(out_device.type()), out_device.index() };
      |                         ^~~
/autofs/nccs-svm1_sw/summit/ums/gen119/nvrapids_21.06_src/cucim_v21.06.00/cpp/plugins/cucim.kit.cuslide/src/cuslide/tiff/ifd.cpp:262:31: error: 'DLContext' was not declared in this scope; did you mean 'ucontext'?
  262 |     out_image_container.ctx = DLContext{ static_cast<DLDeviceType>(out_device.type()), out_device.index() };
      |                               ^~~~~~~~~
      |                               ucontext
/autofs/nccs-svm1_sw/summit/ums/gen119/nvrapids_21.06_src/cucim_v21.06.00/cpp/plugins/cucim.kit.cuslide/src/cuslide/tiff/tiff.cpp: In member function 'bool cuslide::tiff::TIFF::read_associated_image(const cucim::io::format::ImageMetadataDesc*, const cucim::io::format::ImageReaderRegionRequestDesc*, cucim::io::format::ImageDataDesc*, cucim::io::format::ImageMetadataDesc*)':
/autofs/nccs-svm1_sw/summit/ums/gen119/nvrapids_21.06_src/cucim_v21.06.00/cpp/plugins/cucim.kit.cuslide/src/cuslide/tiff/tiff.cpp:718:25: error: 'struct DLTensor' has no member named 'ctx'
  718 |     out_image_container.ctx = DLContext{ static_cast<DLDeviceType>(out_device.type()), out_device.index() };
      |                         ^~~
/autofs/nccs-svm1_sw/summit/ums/gen119/nvrapids_21.06_src/cucim_v21.06.00/cpp/plugins/cucim.kit.cuslide/src/cuslide/tiff/tiff.cpp:718:31: error: 'DLContext' was not declared in this scope; did you mean 'ucontext'?
  718 |     out_image_container.ctx = DLContext{ static_cast<DLDeviceType>(out_device.type()), out_device.index() };

Thanks

[grlee77]

Note: Image processing part (scikit-image compatible APIs using CuPy) is pure Python code and no platform-dependent as long as CuPy is installed. We may want to provide an option to build only one of them (cucim.skimage and cucim.clara).

I think this is already possible, we just don't have a pre-packaged version with only the cucim.skimage component. If you check out the repository, change to the python/cucim folder and run pip install . -v it should install the Python package, but without any of the compiled binaries. Thus cucim.clara would be unavailable, but cucim.skimage should still work since all compilation there is currently at runtime through CuPy. CuPy>=9 would be required at run time, but no other RAPIDS packages would be.

This currently seems to work, but I have not confirmed with the NVIDIA team whether we can continue to guarantee this going forward (we do not currently test this installation method)

[gigony]

@benjha Thanks for sharing the use cases!

Is cuCIM not meant to be build within an existent built from source Rapids environment ?

That's correct. Currently, cuCIM's build doesn't depend on the same version of other RAPIDS components (and other RAPIDS suites) -- That's why it uses specific versions of rmm and DLPack from external Git repositories.

Please refer to the following documents for building cuCIM locally: https://github.com/gigony/cucim/blob/python_unittests/CONTRIBUTING.md

And thanks for the information regarding rmm/DLPack versions. I would update those versions later.

[jakirkham]

Closing as answered. Happy to discuss more questions in new issues as needed