jfcoz / postgresqltuner

Simple script to analyse your PostgreSQL database configuration, and give tuning advice
https://postgresqltuner.pl
GNU General Public License v3.0
2.61k stars 190 forks source link

Does not work on FreeBSD #33

Closed cstdenis closed 5 years ago

cstdenis commented 6 years ago
[INFO]    OS: freebsd Version: 11.2-release-p4 Arch: amd64-freebsd-thread-multi
Use of uninitialized value $os_mem in pattern match (m//) at ./postgresqltuner.pl line 255.
Use of uninitialized value $os_mem in pattern match (m//) at ./postgresqltuner.pl line 256.
[INFO]    OS total memory: 0
[WARN]    Unable to open sysctl vm/overcommit_memory
Use of uninitialized value $overcommit_memory in numeric ne (!=) at ./postgresqltuner.pl line 265.
[BAD]     Memory overcommitment is allowed on the system. This can lead to OOM Killer killing some PostgreSQL process, which will cause a PostgreSQL server restart (crash recovery)
[WARN]    Unable to open sysctl vm/overcommit_ratio
Use of uninitialized value $overcommit_ratio in concatenation (.) or string at ./postgresqltuner.pl line 269.
[INFO]    sysctl vm.overcommit_ratio=
Use of uninitialized value $overcommit_ratio in numeric le (<=) at ./postgresqltuner.pl line 270.
Use of uninitialized value $overcommit_ratio in concatenation (.) or string at ./postgresqltuner.pl line 271.
[BAD]     vm.overcommit_ratio is too small, you will not be able to use more than *RAM+SWAP for applications
Unable to open /sys/block at ./postgresqltuner.pl line 313.

There is no free command, best bet for os_mem is probably a modified version of your darwin way: top -S -n 0

Produces output of

last pid: 92802;  load averages:  0.00,  0.00,  0.00  up 1+20:12:13    22:26:16
56 processes:  2 running, 53 sleeping, 1 waiting

Mem: 910M Active, 160M Inact, 10G Wired, 114G Free
ARC: 7177M Total, 844M MFU, 5842M MRU, 51K Anon, 61M Header, 430M Other
     2984M Compressed, 3704M Uncompressed, 1.24:1 Ratio
Swap: 2048M Total, 2048M Free

The overcommit sysctl is vm.overcommit

Setting bit 0 of the vm.overcommit sysctl causes the virtual memory
     system to return failure to the process when allocation of memory causes
     vm.swap_reserved to exceed vm.swap_total.  Bit 1 of the sysctl enforces
     RLIMIT_SWAP limit (see getrlimit(2)).  Root is exempt from this limit.
     Bit 2 allows to count most of the physical memory as allocatable, except
     wired and free reserved pages (accounted by vm.stats.vm.v_free_target and
     vm.stats.vm.v_wire_count sysctls, respectively).

I don't think there is an equivalent to overcommit_ratio

FreeBSD does not use sysfs, and it's doesn't really give a choice of IO schedulers anyway.

2ge commented 5 years ago

same problem here, running on FreeBSD 11.2-STABLE

postgresqltuner.pl version 1.0.0
Checking if OS commands is available on db1...
[BAD]     Unable to run OS command, report will be incomplete
Connecting to db1:5432 database osdb3 with user osdb3...
[OK]      User used for report have super rights
=====  OS information  =====
[UNKNOWN] Unable to run OS commands on db1. For now you will not have OS information
=====  General instance informations  =====
-----  Version  -----
[OK]      You are using last 10.5
-----  Uptime  -----
[INFO]    Service uptime : 98d 11m 32s
-----  Databases  -----
[INFO]    Database count (except templates): 3
[INFO]    Database list (except templates): postgres osdb3 osdb3_development
-----  Extensions  -----
[INFO]    Number of activated extensions : 2
[INFO]    Activated extensions : plpgsql hstore
[WARN]    Extensions pg_stat_statements is disabled
-----  Users  -----
[OK]      No user account will expire in less than 7 days
[OK]      No user with password=username
[OK]      Password encryption is enabled
-----  Connection information  -----
[INFO]    max_connections: 1000
[INFO]    current used connections: 164 (16.40%)
[INFO]    3 are reserved for super user (0.30%)
[INFO]    Average connection age : 3d 01h 08m 48s
-----  Memory usage  -----
[INFO]    configured work_mem: 4.00 MB
[INFO]    Using an average ratio of work_mem buffers by connection of 150% (use --wmp to change it)
[INFO]    total work_mem (per connection): 6.00 MB
[INFO]    shared_buffers: 8.00 GB
[INFO]    Track activity reserved size : 1011.00 KB
[WARN]    maintenance_work_mem is less or equal default value. Increase it to reduce maintenance tasks time
[INFO]    Max memory usage :
                  shared_buffers (8.00 GB)
                + max_connections * work_mem * average_work_mem_buffers_per_connection (1000 * 4.00 MB * 150 / 100 = 5.86 GB)
                + autovacuum_max_workers * maintenance_work_mem (3 * 64.00 MB = 192.00 MB)
                + track activity size (1011.00 KB)
                = 14.05 GB
[INFO]    effective_cache_size: 4.00 GB
[INFO]    Size of all databases : 20.74 GB
[UNKNOWN] OS total mem unknown : unable to analyse PostgreSQL memory usage
-----  Logs  -----
[OK]      log_hostname is off : no reverse DNS lookup latency
[WARN]    log of long queries is desactivated. It will be more difficult to optimize query performances
[OK]      log_statement=none
-----  Two phase commit  -----
[OK]      Currently no two phase commit transactions
-----  Autovacuum  -----
[OK]      autovacuum is activated.
[INFO]    autovacuum_max_workers: 3
-----  Checkpoint  -----
[WARN]    checkpoint_completion_target(0.5) is low
-----  Disk access  -----
[OK]      fsync is on
[OK]      synchronize_seqscans is on
-----  WAL  -----
-----  Planner  -----
[OK]      costs settings are defaults
[UNKNOWN] Information about rotational/SSD disk is unknown : unable to check random_page_cost and seq_page_cost tuning
[OK]      all plan features are enabled
=====  Database information for database osdb3  =====
-----  Database size  -----
[INFO]    Database osdb3 total size : 20.42 GB
[INFO]    Database osdb3 tables size : 11.30 GB (55.33%)
[INFO]    Database osdb3 indexes size : 9.12 GB (44.67%)
-----  Tablespace location  -----
[OK]      No tablespace in PGDATA
-----  Shared buffer hit rate  -----
[INFO]    shared_buffer_heap_hit_rate: 83.21%
[INFO]    shared_buffer_toast_hit_rate: 99.74%
[INFO]    shared_buffer_tidx_hit_rate: 99.98%
[INFO]    shared_buffer_idx_hit_rate: 99.77%
[OK]      Shared buffer idx hit rate is very good
-----  Indexes  -----
[OK]      No invalid indexes
[WARN]    Some indexes are unused since last statistics: activities.index_activities_on_action average_caches.average_caches_index_rater_rateable average_caches.index_average_caches_on_rateable_type_and_rateable_id average_caches.index_average_caches_on_rater_id comments.index_comments_on_osdbid comments.index_comments_on_user_id companies.index_companies_on_parent_company_id companies_features.index_companies_features_on_company_id companies_features.index_companies_features_on_feature_id featfiles.index_featfiles_on_file_hash featfiles.index_featfiles_on_hash_int feathashes.index_feathashes_on_file_hash feathashes.index_feathashes_on_osdb_movie_hash_id features.index_features_on_plot features.index_features_on_tagline features.index_features_on_title features.index_features_on_tmdb_images features_networks.index_features_networks_on_feature_id friendly_id_slugs.index_friendly_id_slugs_on_slug_and_sluggable_type friendly_id_slugs.index_friendly_id_slugs_on_sluggable_id genres.index_genres_on_tmdbid lit_incomming_localizations.index_lit_incomming_localizations_on_incomming_id lit_incomming_localizations.index_lit_incomming_localizations_on_locale_id lit_incomming_localizations.index_lit_incomming_localizations_on_localization_id lit_incomming_localizations.index_lit_incomming_localizations_on_localization_key_id lit_incomming_localizations.index_lit_incomming_localizations_on_source_id mobility_string_translations.index_mobility_string_translations_on_keys mobility_string_translations.index_mobility_string_translations_on_query_keys mobility_string_translations.index_mobility_string_translations_on_translatable_attribute mobility_text_translations.index_mobility_text_translations_on_keys mobility_text_translations.index_mobility_text_translations_on_translatable_attribute names.index_names_on_imdbid names.index_names_on_tmdb_details osdb_languages.index_osdb_languages_on_iso_639 overall_averages.index_overall_averages_on_rateable_id_and_rateable_type overall_averages.index_overall_averages_on_rateable_type_and_rateable_id rates.index_rates_on_rateable_id_and_rateable_type rates.index_rates_on_rateable_type_and_rateable_id rates.index_rates_on_rater_id rates.index_rates_on_rater_id_man rating_caches.index_rating_caches_on_cacheable_id_and_cacheable_type rating_caches.index_rating_caches_on_cacheable_type_and_cacheable_id roles.index_roles_on_resource_type_and_resource_id seasons.index_seasons_on_tmdbid seasons.index_seasons_on_tvdbid subfilehashes.index_subfilehashes_on_file_hash subfilehashes.index_subfilehashes_on_osdb_subfilehash_id subfiles.index_subfiles_on_featfiles_count subfiles.index_subfiles_on_file_hash subfiles.index_subfiles_on_subfilehashes_count subtitles.index_subtitles_on_osdb_user_id subtitles.index_subtitles_on_translator_id taggings.index_taggings_on_context taggings.index_taggings_on_tag_id taggings.index_taggings_on_taggable_id taggings.index_taggings_on_taggable_id_and_taggable_type_and_context taggings.index_taggings_on_taggable_type taggings.index_taggings_on_tagger_id taggings.index_taggings_on_tagger_id_and_tagger_type taggings.taggings_idx tags.index_tags_on_name thredded_categories.index_thredded_categories_on_messageboard_id_and_slug thredded_categories.thredded_categories_name_ci thredded_messageboard_notifications_for_followed_topics.thredded_messageboard_notifications_for_followed_topics_unique thredded_messageboards.index_thredded_messageboards_on_messageboard_group_id thredded_post_moderation_records.index_thredded_moderation_records_for_display thredded_posts.index_thredded_posts_on_messageboard_id thredded_posts.index_thredded_posts_on_postable_id thredded_posts.thredded_posts_content_fts thredded_private_topics.index_thredded_private_topics_on_last_post_at thredded_topic_categories.index_thredded_topic_categories_on_category_id thredded_topics.index_thredded_topics_for_display thredded_topics.index_thredded_topics_on_last_post_at thredded_topics.thredded_topics_title_fts thredded_user_details.index_thredded_user_details_for_moderations thredded_user_details.index_thredded_user_details_on_latest_activity_at thredded_user_topic_read_states.index_thredded_user_topic_read_states_on_messageboard_id users.index_users_on_translations_count users.index_users_on_type
-----  Procedures  -----
[WARN]    Some user procedures does not have custom cost and rows settings : public.hstore_in public.hstore_out public.hstore_recv public.hstore_send public.hstore_version_diag public.fetchval public.slice_array public.slice public.isexists public.exist public.exists_any public.exists_all public.isdefined public.defined public.delete public.delete public.delete public.hs_concat public.hs_contains public.hs_contained public.tconvert public.hstore public.hstore public.hstore public.hstore_to_json public.hstore_to_json_loose public.hstore_to_jsonb public.hstore_to_jsonb_loose public.hstore public.hstore_to_array public.hstore_to_matrix public.akeys public.avals public.populate_record public.hstore_eq public.hstore_ne public.hstore_gt public.hstore_ge public.hstore_lt public.hstore_le public.hstore_cmp public.hstore_hash public.ghstore_in public.ghstore_out public.ghstore_compress public.ghstore_decompress public.ghstore_penalty public.ghstore_picksplit public.ghstore_union public.ghstore_same public.ghstore_consistent public.gin_extract_hstore public.gin_extract_hstore_query public.gin_consistent_hstore

=====  Configuration advices  =====
-----  checkpoint  -----
[MEDIUM] Your checkpoint completion target is too low. Put something nearest from 0.8/0.9 to balance your writes better during the checkpoint interval
-----  extension  -----
[LOW] Enable pg_stat_statements to collect statistics on all queries (not only queries longer than log_min_duration_statement in logs)
-----  index  -----
[MEDIUM] You have unused indexes in the database since last statistics. Please remove them if they are never use
-----  proc  -----
[LOW] You have custom procedures with default cost and rows setting. Please reconfigure them with specific values to help the planer
-----  report  -----
[URGENT] Please configure your .ssh/config to allow postgresqltuner.pl to connect via ssh to db1 without password authentication. This will allow to collect more system informations
adiSuper94 commented 5 years ago

@jfcoz There seems to be another tool called freecolor that can be installed on freeBSD. It has options to print data in "free" format. If we do a try_load for it and in line 270 add a check for freeBSD and call that instead , it should work.

cstdenis commented 5 years ago

Just installed freecolor, here is it's output for the record. If support is added, the FreeBSD port of postgresqltuner could be adjusted to set it as a dependency.

root@Tsubasa:/home/cstdenis # freecolor
Physical  : [##########################.........] 76%   (99637056/130679480)
Swap      : [###################################] 100%  (2097148/2097148)
root@Tsubasa:/home/cstdenis # freecolor -o
             total       used       free     shared    buffers     cached
Mem:     130679480   31017124   99662356          0          0          0
Swap:      2097148          0    2097148
adiSuper94 commented 5 years ago

@cstdenis Can you test out the fix that I have made?

cstdenis commented 5 years ago
root@Tsubasa:/home/cstdenis # freecolor -m -o
             total       used       free     shared    buffers     cached
Mem:        127616      30846      96770          0          0          0
Swap:         2047          0       2047

Seems to fix the memory issue, but other changes are still needed in line with my comments in the initial report related to overcommit to clear out more bad errors.

root@Tsubasa:/home/cstdenis # ./postgresqltuner.pl --host /tmp --user postgres
postgresqltuner.pl version 1.0.1
Checking if OS commands are available on /tmp...
[OK]      OS command OK
Connecting to /tmp:5432 database template1 with user postgres...
[OK]      User used for report has superuser rights
=====  OS information  =====
[INFO]    OS: freebsd Version: 11.2-release-p10 Arch: amd64-freebsd-thread-multi
[INFO]    OS total memory: 124.63 MB
[UNKNOWN] Unable to read sysctl vm/overcommit_memory
Use of uninitialized value $overcommit_memory in numeric ne (!=) at ./postgresqltuner.pl line 290.
[BAD]     Memory overcommitment is allowed on the system. This can lead to OOM Killer killing some PostgreSQL process, which will cause a PostgreSQL server restart (crash recovery)
[UNKNOWN] Unable to read sysctl vm/overcommit_ratio
Use of uninitialized value $overcommit_ratio in concatenation (.) or string at ./postgresqltuner.pl line 294.
[INFO]    sysctl vm.overcommit_ratio=
Use of uninitialized value $overcommit_ratio in numeric le (<=) at ./postgresqltuner.pl line 295.
Use of uninitialized value $overcommit_ratio in concatenation (.) or string at ./postgresqltuner.pl line 296.
[BAD]     vm.overcommit_ratio is too small, you will not be able to use more than *RAM+SWAP for applications
[INFO]    Running on physical machine
[UNKNOWN] Unable to identify disks
[INFO]    Currently used I/O scheduler(s):
=====  General instance informations  =====
-----  Version  -----
[OK]      You are using latest major 11.3
-----  Uptime  -----
[INFO]    Service uptime:  08h 37m 07s
[WARN]    Uptime is less than 1 day. postgresqltuner.pl result may not be accurate
-----  Databases  -----
[INFO]    Database count (except templates): 4
[INFO]    Database list (except templates): postgres hentai_foundry fanart_central anime_vidcaps
-----  Extensions  -----
[INFO]    Number of activated extensions: 1
[INFO]    Activated extensions: plpgsql
[WARN]    Extensions pg_stat_statements is disabled in database template1
-----  Users  -----
[OK]      No user account will expire in less than 7 days
[OK]      No user with password=username
[OK]      Password encryption is enabled
-----  Connection information  -----
[INFO]    max_connections: 650
[INFO]    current used connections: 30 (4.62%)
[INFO]    3 connections are reserved for super user (0.46%)
[INFO]    Average connection age:  01h 50m 22s
-----  Memory usage  -----
[INFO]    configured work_mem: 512.00 MB
[INFO]    Using an average ratio of work_mem buffers by connection of 150% (use --wmp to change it)
[INFO]    total work_mem (per connection): 768.00 MB
[INFO]    shared_buffers: 40.00 GB
[INFO]    Track activity reserved size: 0.00 B
[INFO]    maintenance_work_mem=8.00 GB
[INFO]    Max memory usage:
                  shared_buffers (40.00 GB)
                + max_connections * work_mem * average_work_mem_buffers_per_connection (650 * 512.00 MB * 150 / 100 = 487.50 GB)
                + autovacuum_max_workers * maintenance_work_mem (3 * 8.00 GB = 24.00 GB)
                + track activity size (0.00 B)
                = 551.50 GB
[INFO]    effective_cache_size: 64.00 GB
[INFO]    Size of all databases: 27.07 GB
[WARN]    shared_buffer is too big for the total databases size, memory is lost
[INFO]    PostgreSQL maximum memory usage: 453145.83% of system RAM
[BAD]     Max possible memory usage for PostgreSQL is more than system total RAM. Add more RAM or reduce PostgreSQL memory
[INFO]    max memory+effective_cache_size is 505732.11% of total RAM
[WARN]    the sum of max_memory and effective_cache_size is too high, the planner can find bad plans if system cache is smaller than expected
-----  Logs  -----
[OK]      log_hostname is off: no reverse DNS lookup latency
[OK]      long queries will be logged
[OK]      log_statement=none
-----  Two phase commit  -----
[OK]      Currently no two phase commit transactions
-----  Autovacuum  -----
[OK]      autovacuum is activated.
[INFO]    autovacuum_max_workers: 3
-----  Checkpoint  -----
[WARN]    checkpoint_completion_target(0.7) is low
-----  Disk access  -----
[OK]      fsync is on
[OK]      synchronize_seqscans is on
-----  WAL  -----
-----  Planner  -----
[WARN]    some cost settings are not the defaults: random_page_cost. This can have bad impacts on performance. Use at your own risk
[UNKNOWN] Information about rotational/SSD disk is unknown: unable to check random_page_cost and seq_page_cost tuning
[BAD]     some plan features are disabled: enable_partitionwise_aggregate,enable_partitionwise_join
=====  Database information for database template1  =====
-----  Database size  -----
[INFO]    Database template1 total size: 10.47 MB
[INFO]    Database template1 tables size: 7.31 MB (69.85%)
[INFO]    Database template1 indexes size: 3.16 MB (30.15%)
-----  Tablespace location  -----
[OK]      No tablespace in PGDATA
-----  Shared buffer hit rate  -----
[INFO]    shared_buffer_heap_hit_rate: 100.00%
[INFO]    shared_buffer_toast_hit_rate: 98.24%
[INFO]    shared_buffer_tidx_hit_rate: 99.85%
[INFO]    shared_buffer_idx_hit_rate: 100.00%
[INFO]    shared buffer idx hit rate too high. You can safely reduce shared_buffer
-----  Indexes  -----
[OK]      No invalid indexes
[OK]      No unused indexes
-----  Procedures  -----
[OK]      No procedures with default costs

=====  Configuration advice  =====
-----  checkpoint  -----
[MEDIUM] Your checkpoint completion target is too low. Put something nearest from 0.8/0.9 to balance your writes better during the checkpoint interval
-----  extension  -----
[LOW] Enable pg_stat_statements in database template1 to collect statistics on all queries (not only queries longer than log_min_duration_statement in logs)
-----  sysctl  -----
[URGENT] set vm.overcommit_memory=2 in /etc/sysctl.conf and run sysctl -p to reload it. This will disable memory overcommitment and avoid postgresql killed by OOM killer.
adiSuper94 commented 5 years ago

@cstdenis FreeBSD has support for virtual memory over commit, using vm.overcommit configuration setting. This setting is configured via /etc/sysctl.conf.

Change 'vm.overcommit: 0 ' to 'vm.overcommit: 1' and try again.

jfcoz commented 5 years ago

Fixed in https://github.com/jfcoz/postgresqltuner/pull/49

cstdenis commented 5 years ago

@adityas71094 I am aware of that, my comments were regarding the script not handling the different sysctl name and generating errors as a result

[UNKNOWN] Unable to read sysctl vm/overcommit_memory
Use of uninitialized value $overcommit_memory in numeric ne (!=) at ./postgresqltuner.pl line 290.
[BAD]     Memory overcommitment is allowed on the system. This can lead to OOM Killer killing some PostgreSQL process, which will cause a PostgreSQL server restart (crash recovery)
[UNKNOWN] Unable to read sysctl vm/overcommit_ratio
Use of uninitialized value $overcommit_ratio in concatenation (.) or string at ./postgresqltuner.pl line 294.
[INFO]    sysctl vm.overcommit_ratio=
Use of uninitialized value $overcommit_ratio in numeric le (<=) at ./postgresqltuner.pl line 295.
Use of uninitialized value $overcommit_ratio in concatenation (.) or string at ./postgresqltuner.pl line 296.