abhishekdwivedi3060
commented
1 month ago Hi @virajshah1996,
The underlying storage provisioning is not in Aerospike Operator's control. Whatever storageClass is defined in the AerospikeCluster storage section, operator requests for PVCs using that storageClass.
It looks like the PVCs provisioned by ebs.csi.aws.com are not clean by default, hence the error not an Aerospike device but not erased - check config or erase device
You need to erase that device using initMethod. Ref this for more info
virajshah1996
this issues faced only when provisioner is ebs.csi.aws.com in storage class yaml otherwise it is working fine when provisioner is kubernetes.io/aws-ebs
below is storage class yaml
allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"ssd-aerospike-gp3-ebs"},"parameters":{"csi.storage.k8s.io/fstype":"ext4","encrypted":"true","iopsPerGB":"50","type":"gp3"},"provisioner":"ebs.csi.aws.com","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"} storageclass.kubernetes.io/is-default-class: "true" creationTimestamp: "2024-09-23T10:29:57Z" name: ssd-aerospike-gp3-ebs resourceVersion: "767939409" uid: 4255df12-99b3-431e-81a3-13ca4437c625 parameters: csi.storage.k8s.io/fstype: ext4 encrypted: "true" iopsPerGB: "50" type: gp3 provisioner: ebs.csi.aws.com reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer
But, pod logs is showing
Defaulted container "aerospike-server" out of: aerospike-server, aerospike-prometheus-exporter, aerospike-init (init) link eth0 state up link eth0 state up in 0 Sep 23 2024 10:22:46 GMT: INFO (as): (as.c:308) <><><><><><><><><><> Aerospike Enterprise Edition build 7.1.0.0 <><><><><><><><><><> Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) logging { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) console { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) context any info Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) namespace flowdb { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) nsup-period 120 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) replication-factor 2 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) storage-engine device { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) device /flowdb/dev/xvdf Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) namespace pipelines { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) nsup-period 120 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) replication-factor 2 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) storage-engine device { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) device /pipelines/dev/xvdf Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) network { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) fabric { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) port 3001 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) heartbeat { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) mesh-seed-address-port aerospike-0-2.aerospike.aerospike 3002 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) mesh-seed-address-port aerospike-0-1.aerospike.aerospike 3002 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) mode mesh Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) port 3002 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) service { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) access-address 10.35.236.109 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) access-port 31416 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) alternate-access-address 10.35.236.109 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) alternate-access-port 31416 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) port 3000 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) security { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) service { Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) cluster-name aerospike Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) feature-key-file /etc/aerospike/secret/features.conf Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) node-id 0a0 Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4316) } Sep 23 2024 10:22:46 GMT: INFO (config): (features_ee.c:273) loaded feature key #912929515 (/etc/aerospike/secret/features.conf) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4348) system file descriptor limit: 1048576, proto-fd-max: 15000 Sep 23 2024 10:22:46 GMT: INFO (hardware): (hardware.c:1989) detected 8 CPU(s), 4 core(s), 1 NUMA node(s) Sep 23 2024 10:22:46 GMT: INFO (config): (cfg.c:4380) node-id a0 Sep 23 2024 10:22:46 GMT: WARNING (os): (os.c:239) failed min-free-kbytes check - min_free_kbytes should be at least 1150976 Sep 23 2024 10:22:46 GMT: WARNING (os): (os.c:260) failed swappiness check - swappiness not set to 0 Sep 23 2024 10:22:46 GMT: WARNING (config): (cfg.c:4764) failed best-practices checks - see 'https://docs.aerospike.com/docs/operations/install/linux/bestpractices/index.html' Sep 23 2024 10:22:46 GMT: INFO (smd): (smd.c:2346) no file '/opt/aerospike/smd/evict.smd' - starting empty Sep 23 2024 10:22:46 GMT: INFO (smd): (smd.c:2346) no file '/opt/aerospike/smd/XDR.smd' - starting empty Sep 23 2024 10:22:46 GMT: INFO (smd): (smd.c:2346) no file '/opt/aerospike/smd/roster.smd' - starting empty Sep 23 2024 10:22:46 GMT: INFO (storage): (drv_common_ee.c:108) {flowdb} peek found fresh device /flowdb/dev/xvdf Sep 23 2024 10:22:46 GMT: INFO (namespace): (namespace_ee.c:249) {flowdb} found no stored data, will cold start Sep 23 2024 10:22:46 GMT: INFO (storage): (drv_common_ee.c:108) {pipelines} peek found fresh device /pipelines/dev/xvdf Sep 23 2024 10:22:46 GMT: INFO (namespace): (namespace_ee.c:249) {pipelines} found no stored data, will cold start Sep 23 2024 10:22:46 GMT: INFO (namespace): (namespace_ee.c:424) {flowdb} beginning cold start Sep 23 2024 10:22:46 GMT: INFO (namespace): (namespace_ee.c:424) {pipelines} beginning cold start Sep 23 2024 10:22:46 GMT: INFO (smd): (smd.c:2346) no file '/opt/aerospike/smd/sindex.smd' - starting empty Sep 23 2024 10:22:46 GMT: INFO (smd): (smd.c:2346) no file '/opt/aerospike/smd/truncate.smd' - starting empty Sep 23 2024 10:22:46 GMT: INFO (drv_ssd): (drv_ssd.c:3514) opened device /flowdb/dev/xvdf: usable size 107374182400, io-min-size 512 Sep 23 2024 10:22:46 GMT: INFO (drv_ssd): (drv_ssd.c:1067) /flowdb/dev/xvdf has 12800 wblocks of size 8388608 Sep 23 2024 10:22:46 GMT: CRITICAL (drv_ssd): (drv_ssd.c:2444) /flowdb/dev/xvdf: not an Aerospike device but not erased - check config or erase device Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:259) SIGUSR1 received, aborting Aerospike Enterprise Edition build 7.1.0.0 os ubuntu22.04 arch x86_64 sha 719892f ee-sha 53fd619 Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:293) si_code SI_TKILL (-6) Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:386) stacktrace: registers: rax 0000000000000000 rbx 0000000000051000 rcx 00000000000000b9 rdx 00007f2a56782f7f rsi 00007f2a56732f81 rdi 00007f2a5efd90f8 rbp 0000000000000003 rsp 00007ffcea542f80 r8 0000000000000000 r9 0000000000000036 r10 0000000000000000 r11 0000000000000000 r12 00007f2a5a000000 r13 00007f2a5e80ee80 r14 00007f2a577ab8a0 r15 00007f2a56732000 rip 000056295f716e40 Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:398) stacktrace: found 15 frames: 0x1458b9 0x145fb3 0x7f2a5f9b3520 0x7f2a5fa079fc 0x7f2a5f9b3476 0x33cf20 0x1e452d 0x1e6025 0x1e769b 0x1e9e20 0xd1330 0xae293 0x7f2a5f99ad90 0x7f2a5f99ae40 0xaf1d5 offset 0x56295f3da000 Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 0: asd(log_stack_trace base/signal.c:389) [0x1458b9] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 1: asd(log_memory base/signal.c:423 sig_trace_and_reraise base/signal.c:183) [0x145fb3] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 2: /lib/x86_64-linux-gnu/libc.so.6() [0x7f2a5f9b3520] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 3: /lib/x86_64-linux-gnu/libc.so.6(pthread_kill) [0x7f2a5fa079fc] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 4: /lib/x86_64-linux-gnu/libc.so.6(raise) [0x7f2a5f9b3476] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 5: asd(cf_log_write_no_return /work/source/cf/src/log.c:612) [0x33cf20] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 6: asd(ssd_read_header storage/drv_ssd.c:2464) [0x1e452d] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 7: asd(ssd_init_synchronous storage/drv_ssd.c:3259) [0x1e6025] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 8: asd(as_storage_init_ssd storage/drv_ssd.c:3863) [0x1e769b] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 9: asd(as_storage_init storage/storage.c:75) [0x1e9e20] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 10: asd(as_run base/as.c:362) [0xd1330] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 11: asd(main base/main.c:38) [0xae293] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 12: /lib/x86_64-linux-gnu/libc.so.6() [0x7f2a5f99ad90] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 13: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main) [0x7f2a5f99ae40] Sep 23 2024 10:22:46 GMT: WARNING (as): (signal.c:405) stacktrace: frame 14: asd(_start) [0xaf1d5]