search

OutsideIT / FireMotD

:fire: Fire Framework Linux MoTD Generator :fire:
https://outsideit.net/firemotd
GNU General Public License v3.0
174 stars 44 forks source link

after upgrade turn FireMotD slow on show up #128

Closed hamwong closed 5 years ago

hamwong commented 5 years ago

Issue Type Slow Bug report / Enhancement Request

Issue Detail After I get upgrade from V.7.05.170509 to v10.03.180725, after around 2-3sec loading, it's show up line by line, each line about 0.8-1.x sec, On previous version, after loading, info will show all info immediately

Expected Behavior I expect same as previous, after loading will show all info at once

Actual Behavior Info show up line by line

How to reproduce Behavior I test on 3 Rpi, 2 Rpi 3 and 1 Rpi 3+, 1xRpi3 on Jessie and 1xRpi3+ on Stretch behavior same, line by line, 1xRpi3 on Jessie with old version V.7.05.17 function normal

I don't know is this is on Design, if it is, is there any options to behavior as previous version?

willemdh commented 5 years ago

@hamwong Hey, you are not the only one complainaing about slowness (https://github.com/OutsideIT/FireMotD/issues/125). On the 600 servers where I'm using it I don't experience a huge difference in speed. (I do notice the difference, but I found it neglectable)

I guess I should keep 1 hardcoded theme, which seems to be much faster on some platforms. I fear for now you better use and older version for now. I'm very busy atm and will be for the coming months. I hope to find some time in the Christmas holidays to improve the speed somehow.

hamwong commented 5 years ago

thanks and apperciated, I will try to fallback then. thanks.

willemdh commented 5 years ago

@hamwong Please try https://github.com/OutsideIT/FireMotD/releases/tag/v11.01.190105 and let me know if things are better once you have a cache file for the theme you are running. My tests say it runs in +- 1,5 seconds with cache and 2,5 without (live)

Also wanted to add that if you omit the CpuUsage from the theme you want to use it will be 1 seconds faster, as the mpstat takes an average oc all cpu's over 1 second.

willemdh commented 5 years ago

@hamwong Please also try multithreading with https://github.com/OutsideIT/FireMotD/releases/tag/v11.02.190106 For example: ./FireMotD -T Eline -MT

This is still an experimental feature though.

hamwong commented 5 years ago

@willemdh Thanks for the update, it works lot better on speed, but there is 2 issue, 1) output get less compare with v7.05.170509

###############################################################################
################# OpenhabianPi-Server ############################# 7.05.170509
###############################################################################
##        Ip = 192.168.0.1
##   Release = Raspbian GNU/Linux 8 (jessie)
##    Kernel = Linux 4.9.35-v7+
##  Platform = Raspberry Pi 3 Model B Rev 1.2
##    Uptime = 2 day(s). 7:36:45
## CPU Usage = 25.94 % avg over 4 cpu(s)  (4 core(s) x 1 socket(s))
##  CPU Load = 1m: 0.12, 5m: 0.19, 15m: 0.25
##    Memory = Free: 0.04GB (4%), Used: 0.90GB (96%), Total: 0.94GB
##      Swap = Free: 0.00GB (1%), Used: 0.09GB (99%), Total: 0.09GB
##      Root = Free: 38.64GB (74%), Used: 13.31GB (26%), Total: 54.77GB
##   Updates = 230 apt-get updates available.
##  Sessions = 1 sessions
## Processes = 202 running processes of 32768 maximum processes
##     Nginx = Version: 1.15.7
##       PHP = Version: 7.2.13, Memory Limit: 128M
###############################################################################

New Version:
###############################################################################
###############  OpenhabianPi-Server  #########################################
###############################################################################
##        Ip = 192.168.0.1
##   Release = Raspbian GNU/Linux 8 (jessie)
##    Kernel = Linux 4.9.35-v7+
##  Platform = Raspberry Pi 3 Model B Rev 1.2
##    Uptime = 2 day(s). 7:53:41
## CPU Usage = 5.51% avg over 4 cpu(s) (4 core(s) x 1 socket(s))
##  CPU Load = 1m: 0.68, 5m: 0.68, 15m: 0.49
##    Memory = Free: 0.40GB (43%), Used: 0.54GB (57%), Total: 0.94GB
##      Swap = Free: 0.00GB (0%), Used: 0.09GB (100%), Total: 0.09GB
##      Root = Free: 38.58GB (74%), Used: 13.37GB (26%), Total: 54.77GB
##   Updates = 230 apt updates available.
##  Sessions = 1 session(s)
## Processes = 204 running processes of 32768 maximum processes
###############################################################################

As you can see, PHP and Nginx was missing

2) if change theme without sudo, I got permission issue `:/opt/FireMotD$ FireMotD -T Eline -MT 2019-01-12 06:15:10,895: FireMotD: Warning: Cache for theme Eline not found in FireMotD cache folder. Generating... touch: cannot touch '/usr/share/firemotd/cache/firemotd-Eline.motd': Permission denied chmod: cannot access '/usr/share/firemotd/cache/firemotd-Eline.motd': No such file or directory /usr/local/bin/FireMotD: line 1602: /usr/share/firemotd/cache/firemotd-Eline.motd: Permission denied (This line repeat a lot)

cat: /usr/share/firemotd/cache/firemotd-Eline.motd: No such file or directory `

3) after run few times of ./FireMotD -T Orange -MT, it fail to run with or without -MT

FireMotD -T Orange -MT 2019-01-12 06:22:56,408: FireMotD: Info: No FireMotD ExportFile detected. Please generate with "sudo .\/FireMotD -S"

I have tried again with -S but nothing show up, and problem won't fix, any suggest?

willemdh commented 5 years ago

Yeah the output is different. I will repopulate the themes with Nginx and Php in all themes. Also I think the /usr/share/firemotd/cache folder should be created with 777 permissions to be able to regenerate the cache files without sudo..

Could you try to:

chmod 777 /usr/share/firemotd/cache
FireMotD -G all -d
sudo FireMotD -S -D all -d

And let me know if that improves your situation?

Also, fyi the -MT switch is higly experimental. The problem is in the Explore CpuUsage which uses mpstat to calculate the cpu avg over 1 second. When you use the -MT switch CpuUsage will be higher, as all Explore function will be executed at the same time. But even without the -MT switch things should be faster then before once you are able to generate the cache file correctly.

hamwong commented 5 years ago

yes, your command mostly work, error msg is gone, but only appear as below, info is gone

###############  OpenhabianPi-Server  #########################################
###############################################################################
###############################################################################

command run as below

2019-01-13 02:57:32,271: FireMotD: Info: Debug mode
2019-01-13 02:57:32,286: FireMotD: Info: FireMotD action save started
2019-01-13 02:57:32,302: FireMotD: Info: Exportfile /usr/share/firemotd/data/FireMotD.json found.
2019-01-13 02:57:32,315: FireMotD: Info: Template type set to all. Overwriting..
2019-01-13 02:57:32,352: FireMotD: Info: Exploring Elasticsearch
2019-01-13 02:57:32,410: FireMotD: Info: Exploring HostIp
2019-01-13 02:57:32,472: FireMotD: Info: Exploring Httpd
2019-01-13 02:57:32,523: FireMotD: Info: Exploring InstallDate
2019-01-13 02:57:32,568: FireMotD: Info: Exploring Kernel
2019-01-13 02:57:32,612: FireMotD: Info: Exploring Kibana
2019-01-13 02:57:32,652: FireMotD: Info: Exploring Logstash
2019-01-13 02:57:32,692: FireMotD: Info: Exploring Mysql
2019-01-13 02:57:32,746: FireMotD: Info: Exploring Nginx
2019-01-13 02:57:32,809: FireMotD: Info: Exploring Php
2019-01-13 02:57:33,020: FireMotD: Info: Exploring Platform
2019-01-13 02:57:33,108: FireMotD: Info: Exploring Postgres
2019-01-13 02:57:33,153: FireMotD: Info: Exploring Release
2019-01-13 02:57:33,225: FireMotD: Info: Exploring Updates
2019-01-13 02:58:08,017: FireMotD: Info: Export system data to "/usr/share/firemotd/data/FireMotD.json" finished.
2019-01-13 02:58:08,030: FireMotD: Info: FireMotD action save finished

further check, fond only 2 theme work, 

[03:09:48] openhabian@OpenhabianPi-Server:~$ FireMotD -P
2019-01-13 03:09:57,898: FireMotD: Info: Starting FireMotD theme presentation
###############################################################################
###############  OpenhabianPi-Server  #########################################
###############################################################################
###############################################################################
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
iiiiiiiiiiiiiii  OpenhabianPi-Server  iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii
ggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OOOOOOOOOOOOOOO  OpenhabianPi-Server  OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
///////////////////////////////////////////////////////////////////////////////
%%%%%%%%%%%%%%%  OpenhabianPi-Server  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
///////////////////////////////////////////////////////////////////////////////
###############################################################################
###############  OpenhabianPi-Server  #########################################
###############################################################################
###############################################################################

                 OpenhabianPi-Server

          Ip = 192.168.0.1
     Release = Raspbian GNU/Linux 8 (jessie)
      Kernel = Linux 4.9.35-v7+
    Platform = Raspberry Pi 3 Model B Rev 1.2
      Uptime = 3 day(s). 4:52:26
   CPU Usage = 10.05% avg over 4 cpu(s) (4 core(s) x 1 socket(s))
    CPU Load = 1m: 1.07, 5m: 0.69, 15m: 0.52
      Memory = Free: 0.39GB (41%), Used: 0.55GB (59%), Total: 0.94GB
        Swap = Free: 0.00GB (0%), Used: 0.09GB (100%), Total: 0.09GB
        Root = Free: 38.53GB (74%), Used: 13.42GB (26%), Total: 54.77GB
     Updates = 230 apt updates available.
    Sessions = 1 session(s)
   Processes = 208 running processes of 32768 maximum processes

##############################################################################
####################  OpenhabianPi-Server ########  11.02.190106
##############################################################################
##            Ip = 192.168.0.1
##       Release = Raspbian GNU/Linux 8 (jessie)
##        Kernel = Linux 4.9.35-v7+
##      Platform = Raspberry Pi 3 Model B Rev 1.2
##        Uptime = 3 day(s). 4:52:29
##     CPU Usage = 11.87% avg over 4 cpu(s) (4 core(s) x 1 socket(s))
##      CPU Load = 1m: 1.07, 5m: 0.69, 15m: 0.52
##        Memory = Free: 0.39GB (41%), Used: 0.55GB (59%), Total: 0.94GB
##          Swap = Free: 0.00GB (0%), Used: 0.09GB (100%), Total: 0.09GB
##          Root = Free: 38.53GB (74%), Used: 13.42GB (26%), Total: 54.77GB
##       Updates = 230 apt updates available
##      Sessions = 1
##     Processes = 208 running processes of 32768 maximum processes
##         nginx = Version: 1.15.7
##           PHP = Version: 7.2.13, Memory Limit: 128M
##############################################################################
###############################################################################
###############  OpenhabianPi-Server  #########################################
###############################################################################
###############################################################################

theme Work Modern, Blanco, Invader, Original

willemdh commented 5 years ago

The themes that work are still hardcoded. Can you post output of for example firemotd-Elastic.motd that should be in /usr/share/firemotd/cache folder? 

sudo ls -la /usr/share/firemotd/cache
total 92
drwxr-xr-x 2 root root 4096 Jan 12 21:42 .
drwxr-xr-x 6 root root 4096 Jan 12 21:42 ..
-rw-rw-rw- 1 root root 7729 Jan 12 21:42 firemotd-Blue.motd
-rw-rw-rw- 1 root root 9135 Jan 12 21:42 firemotd-Digipolis.motd
-rw-rw-rw- 1 root root 9985 Jan 12 21:42 firemotd-Elastic.motd
-rw-rw-rw- 1 root root 9224 Jan 12 21:42 firemotd-Eline.motd
-rw-rw-rw- 1 root root 9605 Jan 12 21:42 firemotd-Gray.motd
-rw-rw-rw- 1 root root 9609 Jan 12 21:42 firemotd-Orange.motd
-rw-rw-rw- 1 root root 7729 Jan 12 21:42 firemotd-Red.motd
-rw-rw-rw- 1 root root 7768 Jan 12 21:42 firemotd-Zebra.motd

Can you also post /usr/share/firemotd/data/FireMotD.json and the output of FireMotD -R Elastic?

hamwong commented 5 years ago

Hi, here you go, really apperciated your fully help

[sudo] password for openhabian:
total 72
drwxrwxrwx 2 root       root       4096 Jan 13 03:12 .
drwxr-xr-x 6 root       root       4096 Jan 12 06:01 ..
-rw-rw-rw- 1 openhabian openhabian 3635 Jan 13 03:12 firemotd-Blanco.motd
-rw-rw-rw- 1 openhabian openhabian 3598 Jan 13 02:27 firemotd-Blue.motd
-rw-rw-rw- 1 openhabian openhabian 4491 Jan 13 02:27 firemotd-Digipolis.motd
-rw-rw-rw- 1 openhabian openhabian 4493 Jan 13 02:27 firemotd-Elastic.motd
-rw-rw-rw- 1 openhabian openhabian 4493 Jan 13 02:27 firemotd-Eline.motd
-rw-rw-rw- 1 openhabian openhabian 4786 Jan 13 02:27 firemotd-Gray.motd
-rw-rw-rw- 1 openhabian openhabian 6701 Jan 13 03:09 firemotd-Modern.motd
-rw-rw-rw- 1 openhabian openhabian 4790 Jan 13 02:28 firemotd-Orange.motd
-rw-rw-rw- 1 openhabian openhabian 3598 Jan 13 02:28 firemotd-Red.motd
-rw-rw-rw- 1 openhabian openhabian 3598 Jan 13 02:28 firemotd-Zebra.motd

firemotd-Elastic.motd content
^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;29m^^[[0m^[[38;5;$
^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;38mO^[[0m^[[38;5;$
^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;29mv^[[0m^[[38;5;$
^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;29mY^[[0m^[[38;5;$
willemdh commented 5 years ago

@hamwong And the output of FireMotD -G Elastic -d and content of /usr/share/firemotd/data/FireMotD.json?

hamwong commented 5 years ago

Excellet, after run FireMotD -G Elastic -d, then FireMotD -t Elastic, it come out now thanks for help me out, sir, sould you mind update your fix way at REAMD page so everyone ulse may have a look when facing issue, I won't mind do it for you.

Really thanks

willemdh commented 5 years ago

@hamwong Feel free to submit a pr to improve the docs. Glad it's working for you now. You can generate caches for all themes with: FireMotD -G all -d

willemdh commented 5 years ago

Took some time to improve the readme a bit. Please check https://github.com/OutsideIT/FireMotD/releases/tag/v11.03.190127