search

389ds / 389-ds-base

The enterprise-class Open Source LDAP server for Linux
https://www.port389.org/
Other
211 stars 92 forks source link

389ds DB size Looks Bigger #6334

Open Hariprasath-3091 opened 1 month ago

Hariprasath-3091 commented 1 month ago

Issue Description We see the DB Size of 389ds looks very big than usual where we suspect some tuning is needed accordingly.

Package Version and Platform:

Steps to Reproduce Steps to reproduce the behavior:

  1. Install the 389ds with above version.
  2. Enable a DB with 3000 records.
  3. you will see the DB size with more than 1 GB

Expected results Expecting the DB size to be less than 300MB

Screenshots image

Additional context We enabled below tuning already as below on the 389ds Instance dsconf slapd-localhost replication set-changelog --suffix "${MYBASE}" --max-age "1d"; dsconf slapd-localhost backend index add --attr sudohost --index-type eq --reindex userroot; dsconf slapd-localhost backend index add --attr ipserviceprotocol --index-type eq --reindex userroot; dsconf slapd-localhost backend index add --attr ou --index-type eq --reindex userroot; dsconf slapd-localhost backend index delete --attr uidNumber userroot; dsconf slapd-localhost backend index add --attr uidNumber --index-type eq --reindex userroot; dsconf slapd-localhost backend index delete --attr gidNumber userroot; dsconf slapd-localhost backend index add --attr gidNumber --index-type eq --reindex userroot; dsconf slapd-localhost backend index delete --attr memberuid userroot; dsconf slapd-localhost backend index add --attr memberuid --index-type eq --reindex userroot; dsconf slapd-localhost security set --tls-client-renegotiation=off; dsconf slapd-localhost config replace nsslapd-idletimeout=60; dsconf slapd-localhost config replace nsslapd-maxdescriptors=524288; dsconf slapd-localhost config replace nsslapd-conntablesize=64000; dsconf slapd-localhost config replace nsslapd-threadnumber=32; dsconf slapd-localhost config replace nsslapd-listenhost=0.0.0.0; dsconf slapd-localhost config replace nsslapd-securelistenhost=0.0.0.0;

tbordaz commented 1 month ago

Hi, I agree that 1Gb looks too much for 3000 entries. Could you provide the output of 'ls -l /var/lib/dirsrv/slapd-localhost/db /var/lib/dirsrv/slapd-localhost/db/*'

progier389 commented 1 month ago

IMHO, That is probably the db cache size that is too big because of the autotuning (autotuning database cache size is based on available system memory), I think that you need to disable the autotuning and configure a small database cache.

Hariprasath-3091 commented 1 month ago

@tbordaz

ls -l /var/lib/dirsrv/slapd-localhost/db /var/lib/dirsrv/slapd-localhost/db/*

ldapmaster01-389ds:/ # ls -l /var/lib/dirsrv/slapd-localhost/db /var/lib/dirsrv/slapd-localhost/db/*
-rw------- 1 dirsrv dirsrv         51 Sep 18 09:09 /var/lib/dirsrv/slapd-localhost/db/DBVERSION
-rw------- 1 dirsrv dirsrv      24576 Sep 18 09:09 /var/lib/dirsrv/slapd-localhost/db/__db.001
-rw------- 1 dirsrv dirsrv   54091776 Sep 23 15:04 /var/lib/dirsrv/slapd-localhost/db/__db.002
-rw------- 1 dirsrv dirsrv 1048567808 Sep 23 15:03 /var/lib/dirsrv/slapd-localhost/db/__db.003
-rw------- 1 dirsrv dirsrv    1081344 Sep 23 15:03 /var/lib/dirsrv/slapd-localhost/db/__db.004
-rw------- 1 dirsrv dirsrv    8863744 Sep 23 15:03 /var/lib/dirsrv/slapd-localhost/db/__db.005
-rw------- 1 dirsrv dirsrv      90112 Sep 23 15:03 /var/lib/dirsrv/slapd-localhost/db/__db.006
-rw------- 1 dirsrv dirsrv   10485760 Sep 23 15:04 /var/lib/dirsrv/slapd-localhost/db/log.0000000026

/var/lib/dirsrv/slapd-localhost/db:
total 1093232
-rw------- 1 dirsrv dirsrv         51 Sep 18 09:09 DBVERSION
-rw------- 1 dirsrv dirsrv      24576 Sep 18 09:09 __db.001
-rw------- 1 dirsrv dirsrv   54091776 Sep 23 15:04 __db.002
-rw------- 1 dirsrv dirsrv 1048567808 Sep 23 15:03 __db.003
-rw------- 1 dirsrv dirsrv    1081344 Sep 23 15:03 __db.004
-rw------- 1 dirsrv dirsrv    8863744 Sep 23 15:03 __db.005
-rw------- 1 dirsrv dirsrv      90112 Sep 23 15:03 __db.006
-rw------- 1 dirsrv dirsrv   10485760 Sep 23 15:04 log.0000000026
drwx------ 2 dirsrv dirsrv       4096 Sep 15 04:52 userroot

/var/lib/dirsrv/slapd-localhost/db/userroot:
total 30740
-rw------- 1 dirsrv dirsrv       51 Sep 18 09:09 DBVERSION
-rw------- 1 dirsrv dirsrv    16384 Sep 15 01:45 aci.db4
-rw------- 1 dirsrv dirsrv   638976 Sep 23 10:37 ancestorid.db4
-rw------- 1 dirsrv dirsrv  1392640 Sep 23 10:37 cn.db4
-rw------- 1 dirsrv dirsrv  2424832 Sep 23 10:37 entryrdn.db4
-rw------- 1 dirsrv dirsrv    90112 Sep 23 10:37 gidNumber.db4
-rw------- 1 dirsrv dirsrv   262144 Sep 23 03:10 givenName.db4
-rw------- 1 dirsrv dirsrv 18243584 Sep 23 15:03 id2entry.db4
-rw------- 1 dirsrv dirsrv    40960 Sep 23 10:37 ipserviceprotocol.db4
-rw------- 1 dirsrv dirsrv   638976 Sep 23 03:10 mail.db4
-rw------- 1 dirsrv dirsrv   614400 Sep 23 14:09 memberuid.db4
-rw------- 1 dirsrv dirsrv    16384 Sep 23 09:10 nsTombstoneCSN.db4
-rw------- 1 dirsrv dirsrv    16384 Sep 23 09:10 nscpEntryDN.db4
-rw------- 1 dirsrv dirsrv    16384 Sep 15 04:52 nsds5ReplConflict.db4
-rw------- 1 dirsrv dirsrv   933888 Sep 23 10:37 nsuniqueid.db4
-rw------- 1 dirsrv dirsrv    16384 Sep 23 10:37 numsubordinates.db4
-rw------- 1 dirsrv dirsrv   548864 Sep 23 10:37 objectclass.db4
-rw------- 1 dirsrv dirsrv   122880 Sep 23 10:37 ou.db4
-rw------- 1 dirsrv dirsrv   352256 Sep 23 10:37 parentid.db4
-rw------- 1 dirsrv dirsrv  3612672 Sep 23 14:16 replication_changelog.db4
-rw------- 1 dirsrv dirsrv   851968 Sep 23 10:37 sn.db4
-rw------- 1 dirsrv dirsrv    16384 Sep 15 00:11 sudohost.db4
-rw------- 1 dirsrv dirsrv   253952 Sep 22 17:11 telephoneNumber.db4
-rw------- 1 dirsrv dirsrv   180224 Sep 23 10:37 uid.db4
-rw------- 1 dirsrv dirsrv   163840 Sep 23 10:37 uidNumber.db4
ldapmaster01-389ds:/ #
tbordaz commented 1 month ago

Like @progier389 said the culprit was the dbcache (__db.003 that is 1Gb by itself). This size is defined by autotuning. You may disable it and tune the attribute 'nsslapd-dbcachesize' (cn=bdb,cn=config,cn=ldbm database,cn=plugins,cn=config). The dbcache does not need to located on disk. In recent versions, for performance benefit, it is located on tmpfs 'nsslapd-db-home-directory=/dev/shm/slapd-instance'

avrp-devops commented 1 week ago

hello,

we have applied changes like below but still we can see DB size is more and not reduced. autosize=0 dbcachesize=268435456

old values autosize= 25 dbcachesize=1048520960

ldapmaster01-389ds:/var/lib/dirsrv/slapd-localhost # du -sh * 4.0K bak 1.3G db 4.0K ldif 16K lock ldapmaster01-389ds:/var/lib/dirsrv/slapd-localhost #

Backup Sizes also getting increased after changes.

85M backup-2024_10_22_12_00_07 85M backup-2024_10_23_00_00_16 85M backup-2024_10_23_12_01_20 964M backup-2024_10_24_00_00_22 964M backup-2024_10_24_12_00_11 964M backup-2024_10_25_00_00_08 964M backup-2024_10_25_12_00_19 964M backup-2024_10_26_00_00_06 964M backup-2024_10_26_12_00_05

As above @Hariprasath-3091 mentioned DB size should be below 300MB.

Regards, Ramana.

mreynolds389 commented 1 week ago

Did you restart the server after adjusting the db cache size? If you didn't the memory mapped files (__db.*) will not be resized. After restarting the server also check the error log to make sure the server is not resizing the db cache.

avrp-devops commented 3 days ago

@mreynolds389 yes we did restart service. verified error logs nothing updated after our update.

Change was implemented on 2024/10/24

logs are there upto 25/08 [25/Aug/2024:05:13:31.907357672 +0000] - INFO - dblayer_copyfile - Copying /var/lib/dirsrv/slapd-localhost/db/userroot/id2entry.db4 to /var/lib/dirsrv/slapd-localhost/bak/reindex_2024-08-25T05:13:31.298272/userroot/id2entry.db4 [25/Aug/2024:05:13:31.912797497 +0000] - INFO - upgradedb_core - userroot: Start upgradedb. [25/Aug/2024:05:13:31.925428355 +0000] - INFO - bdb_instance_start - Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [25/Aug/2024:05:13:31.930465078 +0000] - INFO - check_and_set_import_cache - pagesize: 4096, available bytes 7224991744, process usage 27254784 [25/Aug/2024:05:13:31.934424396 +0000] - INFO - check_and_set_import_cache - Import allocates 2822262KB import cache. [25/Aug/2024:05:13:32.046468711 +0000] - INFO - bdb_import_main - reindex userroot: Index buffering enabled with bucket size 100 [25/Aug/2024:05:13:32.758246282 +0000] - INFO - import_monitor_threads - reindex userroot: Workers finished; cleaning up... [25/Aug/2024:05:13:32.962982323 +0000] - INFO - import_monitor_threads - reindex userroot: Workers cleaned up. [25/Aug/2024:05:13:32.967199842 +0000] - INFO - bdb_import_main - reindex userroot: Cleaning up producer thread... [25/Aug/2024:05:13:32.975413880 +0000] - INFO - bdb_import_main - reindex userroot: Indexing complete. Post-processing... [25/Aug/2024:05:13:32.979184097 +0000] - INFO - bdb_import_main - reindex userroot: Generating numsubordinates (this may take several minutes to complete)... [25/Aug/2024:05:13:32.982864914 +0000] - INFO - bdb_import_main - reindex userroot: Generating numSubordinates complete. [25/Aug/2024:05:13:33.002692705 +0000] - INFO - bdb_get_nonleaf_ids - reindex userroot: Gathering ancestorid non-leaf IDs... [25/Aug/2024:05:13:33.007153426 +0000] - INFO - bdb_get_nonleaf_ids - reindex userroot: Finished gathering ancestorid non-leaf IDs. [25/Aug/2024:05:13:33.027834721 +0000] - ERR - bdb_ancestorid_new_idl_create_index - Nothing to do to build ancestorid index [25/Aug/2024:05:13:33.035566956 +0000] - INFO - bdb_ancestorid_new_idl_create_index - reindex userroot: Created ancestorid index (new idl). [25/Aug/2024:05:13:33.042921390 +0000] - INFO - bdb_import_main - reindex userroot: Flushing caches... [25/Aug/2024:05:13:33.050737426 +0000] - INFO - bdb_import_main - reindex userroot: Closing files... [25/Aug/2024:05:13:33.137849026 +0000] - INFO - bdb_import_main - reindex userroot: Reindexing complete. Processed 1 entries in 1 seconds. (1.00 entries/sec) [25/Aug/2024:05:13:33.163450844 +0000] - INFO - dblayer_copyfile - Copying /var/lib/dirsrv/slapd-localhost/db/log.0000000001 to /var/lib/dirsrv/slapd-localhost/bak/reindex_2024-08-25T05:13:31.298272/log.0000000001 [25/Aug/2024:05:13:33.207443646 +0000] - INFO - dblayer_copyfile - Copying /var/lib/dirsrv/slapd-localhost/db/DBVERSION to /var/lib/dirsrv/slapd-localhost/bak/reindex_2024-08-25T05:13:31.298272/DBVERSION [25/Aug/2024:05:13:33.215393282 +0000] - INFO - bdb_pre_close - All database threads now stopped [25/Aug/2024:05:13:33.902578340 +0000] - NOTICE - config_set_port - Non-Secure Port Disabled [25/Aug/2024:05:13:33.940621015 +0000] - INFO - util_get_hardware_threads - Automatically configuring 16 threads

Regards, Ramana.