search

heterodb / pg-strom

PG-Strom - Master development repository
http://heterodb.github.io/pg-strom/
Other
1.27k stars 163 forks source link

[VTJ-JP] RHEL 8.8 + PostgreSQL 15の構成でheterodb-extraアップデート後にPG-StromがGPUパスを適切に認識しない #770

Closed ytooyama closed 1 month ago

ytooyama commented 1 month ago

マシン

P.S. 環境固有の問題かもしれません。

PG-Strom ソース版

ほかのソフトウェア

]$ rpm -q heterodb-extra
heterodb-extra-5.4-1.el8.x86_64
]$ rpm -q postgresql15-server postgresql15-devel
postgresql15-server-15.7-1PGDG.rhel8.x86_64
postgresql15-devel-15.7-1PGDG.rhel8.x86_64
]$ rpm -q cuda-tools-12-2 cuda-drivers
cuda-tools-12-2-12.2.1-1.x86_64
cuda-drivers-535.86.10-1.x86_64

環境

]$ uname -a
Linux kujira.vtj 4.18.0-477.55.1.el8_8.x86_64 #1 SMP Thu Apr 4 09:26:55 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux
]$ sudo dkms status
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/nvidia-fs/2.17.3/source/dkms.conf)
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/nvidia-fs/2.17.3/source/dkms.conf)
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/nvidia-fs/2.17.3/source/dkms.conf)
nvidia-fs/2.17.3, 4.18.0-477.43.1.el8_8.x86_64, x86_64: installed
nvidia-fs/2.17.3, 4.18.0-477.51.1.el8_8.x86_64, x86_64: installed
nvidia-fs/2.17.3, 4.18.0-477.55.1.el8_8.x86_64, x86_64: installed
nvidia/535.86.10, 4.18.0-477.43.1.el8_8.x86_64, x86_64: installed
nvidia/535.86.10, 4.18.0-477.51.1.el8_8.x86_64, x86_64: installed
nvidia/535.86.10, 4.18.0-477.55.1.el8_8.x86_64, x86_64: installed

ログ

<no GPUs>という出力が見られる。

01:50:05.666 UTC [46842] LOG:  PG-Strom: GPU0 Tesla V100-SXM2-16GB (80 SMs; 1530MHz, L2 6144kB), RAM 15.77GB (4096b>
01:50:05.673 UTC [46842] LOG:  [0000:b2:00:1] mlx5_1 (MT4123) --> <no GPUs> [dist=-1]
01:50:05.673 UTC [46842] LOG:  [0000:b2:00:0] mlx5_0 (MT4123) --> <no GPUs> [dist=-1]
01:50:05.673 UTC [46842] LOG:  [0000:1d:00:0] nvme0 (VO002000KWJSF) --> <no GPUs> [dist=-1]
01:50:05.673 UTC [46842] LOG:  [0000:20:00:0] nvme3 (VO002000KWJSF) --> <no GPUs> [dist=-1]
01:50:05.674 UTC [46842] LOG:  [0000:1f:00:0] nvme2 (VO002000KWJSF) --> <no GPUs> [dist=-1]
01:50:05.674 UTC [46842] LOG:  [0000:1e:00:0] nvme1 (VO002000KWJSF) --> <no GPUs> [dist=-1]
01:50:05.823 UTC [46842] LOG:  redirecting log output to logging collector process
01:50:05.823 UTC [46842] HINT:  Future log output will appear in directory "log".

クエリをすると、GPU-Direct SQL: disabled

testdb=# EXPLAIN ANALYZE SELECT count(*) FROM arrow_uriage1;
                                                              QUERY PLAN

--------------------------------------------------------------------------------------------------------------------
------------------
 Aggregate  (cost=1662.51..1662.52 rows=1 width=8) (actual time=376.899..376.901 rows=1 loops=1)
   ->  Custom Scan (GpuPreAgg) on arrow_uriage1  (cost=1662.50..1662.51 rows=1 width=8) (actual time=376.892..376.89
3 rows=1 loops=1)
         GPU Projection: pgstrom.nrows()
         referenced:
         file0: /opt/nvme/test/uriage1.arrow (read: 0B, size: 255.11MB)
         GPU-Direct SQL: disabled (GPU-0; ntuples=10000000)
 Planning Time: 4.874 ms
 Execution Time: 377.148 ms
(8 rows)

//拡張が古いのを発見
testdb=# \dx
                                   List of installed extensions
   Name   | Version |   Schema   |                          Description
----------+---------+------------+----------------------------------------------------------------
 pg_strom | 5.0     | public     | PG-Strom - Extension for Big-Data Processing using GPU/DPU/SPU
 plpgsql  | 1.0     | pg_catalog | PL/pgSQL procedural language
(2 rows)

//最新のPG-Stromをソースインストールしつつ
//一度EXTENSIONを導入し直し
testdb=# DROP EXTENSION pg_strom CASCADE;
NOTICE:  drop cascades to 2 other objects
DETAIL:  drop cascades to foreign table arrow_uriage1
drop cascades to foreign table arrow_uriage2
DROP EXTENSION
testdb=# create extension pg_strom;

//新しいのになった
testdb=# \dx
                                   List of installed extensions
   Name   | Version |   Schema   |                          Description
----------+---------+------------+----------------------------------------------------------------
 pg_strom | 5.1     | public     | PG-Strom - big-data processing acceleration using GPU and NVME
 plpgsql  | 1.0     | pg_catalog | PL/pgSQL procedural language
(2 rows)

//データを再度登録
IMPORT FOREIGN SCHEMA arrow_uriage1
  FROM SERVER arrow_fdw
  INTO public
OPTIONS (file '/opt/nvme/test/uriage1.arrow');

IMPORT FOREIGN SCHEMA arrow_uriage2
  FROM SERVER arrow_fdw
  INTO public
OPTIONS (file '/opt/nvme/test/uriage2.arrow');

//まだ「GPU-Direct SQL: disabled」
testdb=# EXPLAIN ANALYZE SELECT count(*) FROM arrow_uriage1;
                                                              QUERY PLAN

--------------------------------------------------------------------------------------------------------------------
------------------
 Aggregate  (cost=1662.51..1662.52 rows=1 width=8) (actual time=348.031..348.034 rows=1 loops=1)
   ->  Custom Scan (GpuPreAgg) on arrow_uriage1  (cost=1662.50..1662.51 rows=1 width=8) (actual time=348.021..348.02
4 rows=1 loops=1)
         GPU Projection: pgstrom.nrows()
         referenced:
         file0: /opt/nvme/test/uriage1.arrow (read: 0B, size: 255.11MB)
         GPU-Direct SQL: disabled (GPU-0; ntuples=10000000)
 Planning Time: 0.243 ms
 Execution Time: 348.256 ms
(8 rows)

// GDSはNVMe: Supported
$ sudo ./gdscheck -p
 GDS release version: 1.7.1.12
 nvidia_fs version:  2.17 libcufile version: 2.12
 Platform: x86_64
 ============
 ENVIRONMENT:
 ============
 =====================
 DRIVER CONFIGURATION:
 =====================
 NVMe               : Supported
 NVMeOF             : Unsupported
 SCSI               : Unsupported
 ScaleFlux CSD      : Unsupported
 NVMesh             : Unsupported
 DDN EXAScaler      : Unsupported
 IBM Spectrum Scale : Unsupported
 NFS                : Unsupported
 WekaFS             : Unsupported
 Userspace RDMA     : Unsupported
 --Mellanox PeerDirect : Disabled
 --rdma library        : Not Loaded (libcufile_rdma.so)
 --rdma devices        : Not configured
 --rdma_device_status  : Up: 0 Down: 0
 =====================
 CUFILE CONFIGURATION:
 =====================
 properties.use_compat_mode : true
 properties.force_compat_mode : false
 properties.gds_rdma_write_support : true
 properties.use_poll_mode : false
 properties.poll_mode_max_size_kb : 4
 properties.max_batch_io_size : 128
 properties.max_batch_io_timeout_msecs : 5
 properties.max_direct_io_size_kb : 16384
 properties.max_device_cache_size_kb : 131072
 properties.max_device_pinned_mem_size_kb : 33554432
 properties.posix_pool_slab_size_kb : 4 1024 16384
 properties.posix_pool_slab_count : 128 64 32
 properties.rdma_peer_affinity_policy : RoundRobin
 properties.rdma_dynamic_routing : 0
 fs.generic.posix_unaligned_writes : false
 fs.lustre.posix_gds_min_kb: 0
 fs.weka.rdma_write_support: false
 fs.gpfs.gds_write_support: false
 profile.nvtx : false
 profile.cufile_stats : 0
 miscellaneous.api_check_aggressive : false
 execution.max_io_threads : 4
 execution.max_io_queue_depth : 128
 execution.parallel_io : true
 execution.min_io_threshold_size_kb : 8192
 execution.max_request_parallelism : 4
 properties.force_odirect_mode : false
 properties.prefer_iouring : false
 =========
 GPU INFO:
 =========
 GPU index 0 Tesla V100-SXM2-16GB bar:1 bar size (MiB):16384 supports GDS, IOMMU State: Disabled
 ==============
 PLATFORM INFO:
 ==============
 IOMMU: disabled
 Platform verification succeeded
kaigai commented 1 month ago

すいません。ライセンスファイルを書き換えてませんか？これ。

$ cat /etc/heterodb.license
IAAGRjmOY+xZOb0SwdZgE+T/jcux0xs1e70Ol9g6//ZDbeC2D6uLwAHNJxBljl2kMSwC2pSzxmgSJuhgi3Xyw1t9iLenma6jFC3N8wzUYPh+6gZdtRFB8cQN+m9jq5NAhzCFj1Z8VRRzvAYKsppA4/mF8udodwFSLSMEXfMijyW4Hi0fHdsS1SU3BtzgpYssjvwzuECsaRnvgHAsb3GsZbVFQAr3xLyWG7riQt74zProXaDecOULojdry4dNhAfDzrcfB7xhNrUX6pZWB+VWmpkIVNreWb43TOmT6im2Q/IKwQ/62YyLAEO1FF3fo1z0BOHGo2fg7V7sNNw4CqbYXMgT=
----
VERSION:2
SERIAL_NR:VTJ-TRIAL-KUJIRA
ISSUED_AT:2023-01-20
EXPIRED_AT:2099-12-31
GPU_UUID:GPU-de13daa5-7c25-e2e5-5971-d08c77103b85

暗号化部分を復号化すると

$ ./license_tool -q /etc/heterodb.license
{ "version" : 2, "serial_nr" : "VTJ-TRIAL-KUJIRA", "issued_at" : "2023-01-20", "expired_at" : "2099-12-31", "nr_gpus" : 3, "gpus" : [ { "uuid" : "GPU-c8205e71-b22d-00d8-7c56-6a75d8e696fc" }, { "uuid" : "GPU-e17b7308-b434-b2e5-1678-218a06378a9e" }, { "uuid" : "GPU-7f67e09e-a284-dbeb-3023-435d325abc73" } ]}

別のGPU x3個のUUIDが出てきますが、当然、これらは Kujira には搭載されていないので、 heterodb-extraはこれを認識しません。 したがって、認識しないGPUとの最短距離といったものは存在しないので、これは正常動作です。

kaigai commented 1 month ago

PostgreSQLの起動ログを見ても、見知らぬGPU-UUIDが表示されているので、ライセンスファイルとGPUが一致していないだけです。

May 29 15:49:49 kujira.vtj postmaster[408777]: 2024-05-29 15:49:49.552 UTC [408777] LOG:  HeteroDB Extra module loaded [api_version=20240418,cufile=on,nvme_strom=off,githash=3ffc65428c07bb3c9d0e5c75a2973389f91dfcd4]
May 29 15:49:49 kujira.vtj postmaster[408777]: 2024-05-29 15:49:49.552 UTC [408777] LOG:  HeteroDB License: { "version" : 2, "serial_nr" : "VTJ-TRIAL-KUJIRA", "issued_at" : "2023-01-20", "expired_at" : "2099-12-31", "nr_gpus" : 3, "gpus" : [ { "uuid" : "GPU-c8205e71-b22d-00d8-7c56-6a75d8e696fc" }, { "uuid" : "GPU-e17b7308-b434-b2e5-1678-218a06378a9e" }, { "uuid" : "GPU-7f67e09e-a284-dbeb-3023-435d325abc73" } ]}
May 29 15:49:49 kujira.vtj postmaster[408777]: 2024-05-29 15:49:49.552 UTC [408777] LOG:  PG-Strom version 5.1.2 built for PostgreSQL 15 (githash: ab327b61ffe74f482297f888c4df81658245dcfa)