Using kdb with PyTorch wheels

[jithunnair-amd]

PyTorch wheels are self-contained wrt ROCm (and hence MIOpen) libraries, bundling them as part of the wheel.

When running workloads with the PyTorch wheel, we observe slow warmup times and see messages such as: MIOpen(HIP): Warning [SQLiteBase] Missing system database file: gfx900_64.kdb Performance may degrade. Please follow instructions to install: https://github.com/ROCmSoftwarePlatform/MIOpen#installing-miopen-kernels-package

However, even after installing the kdb files, we still see the same error. Is it possible for the user to use kdb with PyTorch wheels?

[atamazov]

@jithunnair-amd I recommend assigning the "urgency" and/or "value" labels to the ticket to inform us how important is this for you.

[atamazov]

@jithunnair-amd Question. "Installation of kdb files" means installation of the System Binary cache. As far as I understand, you would like MIOpen to read it. Would you like PyTotch wheels to read/write the User Binary cache in $HOME/.cache/miopen?

[Sfinx]

same here at v5.2.0. it complains while the file is definitely present:

rus@Shiva:~$ dpkg --get-selections |grep miope
miopen-hip-dev5.2.0             install
miopen-hip5.2.0                 install
miopenkernels-gfx900-56kdb          install
miopenkernels-gfx900-64kdb          install

rus@Shiva:~$ l /opt/rocm/share/miopen/db/gfx900_56.kdb
-rw-r--r-- 1 root root 530513920 июн 28 06:43 /opt/rocm/share/miopen/db/gfx900_56.kdb

rus@Shiva:~$ MIOPEN_SYSTEM_DB_PATH=/opt/rocm/share/miopen/db python3 test.py 

MIOpen(HIP): Warning [SQLiteBase] Missing system database file: /opt/rocm/share/miopen/db/gfx900_56.kdb Performance may degrade. Please follow instructions to install: https://github.com/ROCmSoftwarePlatform/MIOpen#installing-miopen-kernels-package

[atamazov]

PyTorch wheels are self-contained

This is the root cause of the problem. For PyTorch wheels, MIOpen embeds kernel databases in order to keep the wheel self-contained. The wheel you are using seems missing embedded image of gfx900_64.kdb. Adding it ti the file system won't help. The wheel needs to be rebuilt with MIOPEN_EMBED_DB cmake variable updated.

/cc @JehandadKhan

[jithunnair-amd]

This PR enables usage of kdbs with PyTorch wheels: https://github.com/ROCmSoftwarePlatform/builder/pull/16, by using the default MIOpen instead of rebuilding an embedded version of MIOpen from source.

[ppanchad-amd]

@jithunnair-amd Is this fixed with latest ROCm 6.0.2 (HIP 6.0.32831)? If resolved, please close ticket. Thanks!