threefoldtech / home

Starting point for the threefoldtech organization

https://threefold.io

Apache License 2.0

9 stars 4 forks source link

better logging & alerthandling [development_logalert] #521

Closed despiegk closed 4 years ago

despiegk commented 4 years ago

checklist

[X] all logs & alerts in redis
[X] logs are configurable in the jumpscale config file
[x] cmdline log client which allows to see logs and filter
- [x] filter running processes (use pid)
- [x] filter based on context, time
[x] alert client which allows to see logs & filter
- [x] time
- [x] full text (part of alert, any text)
- [x] pid
- [x] category
[x] see alerts in web ui of 3bot (add traceback text to alert).
[x] when we start threebotserver no debug logs & no debug pane
[x] doc, so we know what logging there is and how to use

Deliverable by end of Jan

abom commented 4 years ago

Added process ID to logs, and --pid options to js_logs.

Now we push log to redis as a json (not a formatted log string).

Repos: jumpscaleX_core and jumpcaleX_libs

https://github.com/threefoldtech/jumpscaleX_core/tree/development_logalert https://github.com/threefoldtech/jumpscaleX_libs/tree/development_logalert

abom commented 4 years ago

Updated RedisAlertHandler:

Added print, print_list and refactored find to accept pid and time, also more normalization to cat and message inputs.
Updated alerts client not to modify j.tools.alerthandler own redis connection.
Refactored and delegated all calls of js_alerts to alerts client.
Fixed logging client and refactored js_logs to use context object in the click app.

Changes still can be found here and here.

Until documentation are finished, anyone can test js_logs and js_alerts commands, some examples:

Assuming a kosmos or a threebot session is started locally, to tail logs, you can simply do

js_logs tail

To dump to a file, just use dump command:

js_logs dump --path /a/log.ansi

Control + C to stop.

For js_alerts, for example, you can find alert that contain a message (or a public message) of "aa" using:

js_alerts find --message aa

To list all:

js_alerts list

abom commented 4 years ago

[x] Need to add some helper methods to get a formatted trackback from alert objects, like j.core.tools.traceback_format or so, they're now stored as jumpscale.alerthandler.alert.traceback and jumpscale.alerthandler.alert.tracebackitem, see RedisAlertHandler.py

waleedhammam commented 4 years ago

Added some fixes at: https://github.com/threefoldtech/jumpscaleX_core/commit/2451ef6f98aa1a42395a732ea082437a7b8d721b

todo

define applications locations as logs now defined per application j.application.start
check alerts

waleedhammam commented 4 years ago

Related commits https://github.com/threefoldtech/jumpscaleX_libs/commit/9fe73a40c4e1e42e37b7cc5d84b10fa74b69f0e4, https://github.com/threefoldtech/jumpscaleX_core/commit/992c19e9451e7ae5f1cacd4bee2fa2924a0e1f44, https://github.com/threefoldtech/jumpscaleX_core/commit/ef9f0950c04c97a13a3db03eaa249219589c28a3, https://github.com/threefoldtech/jumpscaleX_core/commit/88baf766bdb7c26cdbdcd8da4ebbe2003e2b7997

Logger README https://github.com/threefoldtech/jumpscaleX_core/tree/development_logalert/docs/logger

Changelog

fix js_logs
some fixes in logger client
fixes in alerts

waleedhammam commented 4 years ago

Added alerts readme: https://github.com/threefoldtech/jumpscaleX_core/blob/development_logalert/docs/alerts/README.md

waleedhammam commented 4 years ago

Commit https://github.com/threefoldtech/jumpscaleX_threebot/commit/1d2880cd34e9778774ba3cbe620c8a19cf47e91e

Update alerta_ui to get the tracebacks
multiple tracebacks will be in tabs

Screenshot from 2020-02-06 16-28-24

waleedhammam commented 4 years ago

Commit: https://github.com/threefoldtech/jumpscaleX_core/commit/5100695ca61634c19fdcb4f0862aa184d7451d1f

Changelog

when we start threebotserver no debug logs & no debug pane

waleedhammam commented 4 years ago

https://github.com/threefoldtech/jumpscaleX_core/blob/development_logalert/docs/logger/README.md

Due to new loggers, when you start 3bot server you won't have logs nor pane

to access logs use logger client or js_logs js_logs --appname threebotserver tail or js_logs --appname worker_1 tail

default appname is init and it's available when you start kosmos just js_logs tail

If you need to specify appname in your application, we should use j.application.start("myappname")

hossnys commented 4 years ago

verified on branch unstable tail init logs :

3BOTDEVEL:3bot:jumpscaleX_libs_extra: js_logs tail
Tue 11 11:10:38 Schema.py            - 109 - schema                             : L:
Tue 11 11:10:38 ThreebotServer.py    - 274 - threebotserver_default             : START:zerobot.myjobs_ui
Tue 11 11:10:38 GedisServer.py       - 116 - gedisserver_threebot               : actor_add:%s:%s
--DATA-----------------------
  /sandbox/code/github/threefoldtech/jumpscaleX_threebot/ThreeBotPackages/zerobot/myjobs/actors/myjobs.py
-----------------------------

Tue 11 11:10:38 ThreebotServer.py    - 274 - threebotserver_default             : START:zerobot.oauth2
Tue 11 11:10:38 Schema.py            -  96 - schema                             : load schema
--DATA-----------------------
  @url = jumpscale.oauth_proxy.client
  0 : name** = "main" (S)
  1 : url = "" (S)
  2 : verify_key = (S)
  3 : providers = (LS)
-----------------------------

Tue 11 11:10:38 Schema.py            - 109 - schema                             : L:@url = jumpscale.oauth_proxy.client
Tue 11 11:10:38 Schema.py            - 109 - schema                             : L:0 : name** = "main" (S)
Tue 11 11:10:38 Schema.py            - 109 - schema                             : L:1 : url = "" (S)
Tue 11 11:10:38 Schema.py            - 109 - schema                             : L:2 : verify_key = (S)
Tue 11 11:10:38 Schema.py            - 109 - schema                             : L:3 : providers = (LS)
Tue 11 11:10:38 Schema.py            - 109 - schema                             : L:
Tue 11 11:10:38 BCDB.py              - 599 - bcdb                               : load model:jumpscale.oauth_proxy.client
Tue 11 11:10:38 BCDBModel.py         -  99 - bcdbmodel                          : generate schema index:jumpscale.oauth_proxy.client
Tue 11 11:10:38 Jinja2.py            - 136 - jinja2                             : python code render:/sandbox/var/codegen/bcdbindex_jumpscale_oauth_proxy_client_7b2851b6dba0f98071a96b72279db801.py
Tue 11 11:10:38 071a96b72279db801.py -  13 - t_7b2851b6dba0f98071a96b72279db801 : init index:jumpscale.oauth_proxy.client
Tue 11 11:10:38 Jinja2.py            - 136 - jinja2                             : python code render:/sandbox/var/codegen/schema_jumpscale_oauth_proxy_client_true.py
Tue 11 11:10:38 ThreebotServer.py    - 274 - threebotserver_default             : START:zerobot.packagemanager
Tue 11 11:10:38 GedisServer.py       - 116 - gedisserver_threebot               : actor_add:%s:%s
--DATA-----------------------
  /sandbox/code/github/threefoldtech/jumpscaleX_threebot/ThreeBotPackages/zerobot/packagemanager/actors/package_manager.py
-----------------------------

tail specified application loga e.g: workers_2

3BOTDEVEL:3bot:jumpscaleX_libs_extra: js_logs --appname workers_2 tail
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:3 : time_stop = 0 (T)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:4 : state** = "NEW,ERROR,OK,RUNNING,DONE" (E)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:5 : error_cat** = "NA,TIMEOUT,CRASH,HALTED,ERROR"  (E)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:6 : timeout = 0
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:7 : action_id** = 0 (I)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:8 : kwargs = (dict)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:9 : result = "" (json)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:10: error = (dict)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:11: die = false (B)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:12: dependencies = (LI)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:13: return_queues = (LS)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:14: debug = False (B)
Tue 11 10:40:04 Schema.py            - 109 - schema                             : L:
Tue 11 10:40:04 BCDB.py              - 599 - bcdb                               : load model:jumpscale.myjobs.job
Tue 11 10:40:04 BCDBModel.py         -  99 - bcdbmodel                          : generate schema index:jumpscale.myjobs.job
Tue 11 10:40:04 Jinja2.py            - 136 - jinja2                             : python code render:/sandbox/var/codegen/bcdbindex_jumpscale_myjobs_job_cf9487f9b4afed0462022ac4dac605f0.py
Tue 11 10:40:04 462022ac4dac605f0.py -  13 - b_cf9487f9b4afed0462022ac4dac605f0 : init index:jumpscale.myjobs.job
Tue 11 10:40:04 Jinja2.py            - 136 - jinja2                             : python code render:/sandbox/var/codegen/schema_jumpscale_myjobs_worker_true.py
Tue 11 10:40:04 MyJobsFactory.py     - 129 - myjobsfactory                      : worker in process for tmux: 2
Tue 11 10:40:04 DBSQLite.py          -  30 - dbsqlite                           : EXISTING SQLITEDB in /sandbox/var/bcdb/myjobs/sqlite_stor.db
Tue 11 10:40:04 BCDB.py              -  56 - bcdb                               : sqlite file is in readonly mode for: 'myjobs'
Tue 11 10:40:04 BCDB.py              - 435 - bcdb                               : ** START DATA PROCESSOR FOR :myjobs
Tue 11 10:40:04 BCDB.py              -  71 - bcdb                               : BCDB INIT DONE:myjobs
Tue 11 10:40:04 MyWorkerProcess.py   -  87 - myworkerprocess                    : action: get
--DATA-----------------------
  ## jumpscale.myjobs.worker
  id:2
   - name                : w2
   - timeout             : 3,600
   - time_start          : 2020/02/11 10:40:04
   - last_update         : 2020/02/11 10:40:04
   - current_job         : -
   - error               :
   - state               : WAITING
   - pid                 : 0
   - halt                : False
   - type                : TMUX
   - debug               : False
   - nr                  : 2
-----------------------------

tail specified application loga e.g: threebotserver and search for some text in logs

3BOTDEVEL:3bot:~: js_logs -a threebotserver find --message "connection error"                                                                                            
Tue 11 10:40:04 RedisServer.py       -  85 - redisserver                        : connection error: Error while reading from socket: ('Connection closed by server.',)   
Tue 11 10:40:04 RedisServer.py       -  85 - redisserver                        : connection error: Error while reading from socket: ('Connection closed by server.',)   
Tue 11 10:40:04 RedisServer.py       -  85 - redisserver                        : connection error: Error while reading from socket: ('Connection closed by server.',)
Tue 11 10:40:04 RedisServer.py       -  85 - redisserver                        : connection error: Error while reading from socket: ('Connection closed by server.',)
Tue 11 10:40:04 RedisServer.py       -  85 - redisserver                        : connection error: Error while reading from socket: ('Connection closed by server.',)
Tue 11 10:40:04 RedisServer.py       -  85 - redisserver                        : connection error: Error while reading from socket: ('Connection closed by server.',)
Tue 11 10:40:04 RedisServer.py       -  85 - redisserver                        : connection error: Error while reading from socket: ('Connection closed by server.',)
Tue 11 10:40:11 RedisServer.py       -  85 - redisserver                        : connection error: Error while reading from socket: ('Connection closed by server.',)

Screenshot from 2020-02-11 12-52-59

Screenshot from 2020-02-11 12-53-40

search for a message in alerts using alerts client


JSX> redis_addr = "127.0.0.1" 
   2 redis_port = 6379 
   3 redis_secret = "123"                                                                                                                                                                                          
JSX> client = j.clients.alerts.get("js_alerts")                                                                                                                                                                    
JSX> client.find(message="samir")                                                                                                                                                                                  
[## jumpscale.alerthandler.alert
 - alert_id            : 0
 - identifier          : c496659b50e94f69b178c6e9e63c9067
 - alert_type          : EVENT_SYSTEM
 - level               : 50
 - message             : EXCEPTION: 
    NameError("name 'samir' is not defined",)
 - message_pub         : 
 - cat                 : 
 - count               : 2
 - status              : REOPEN
 - time_first          : 2020/02/11 10:41:41
 - time_last           : 2020/02/11 12:29:45
 - support_trace       : []
 - events              :
    - threebot_name: me    - process_ids: ['3,959', '5,060']    - code_path: <stdin>    - code_line:     - code_line_nr: 1    - count: 2    - time_first: 2020/02/11    - time_last: 2020/02/11    - trace:     - d
ata: null - tracebacks          :
    - threebot_name: me    - process_id: 3,959    - items: []    - threebot_name: me    - process_id: 5,060    - items: [] - logs                :
    - threebot_name: me    - app_name: init    - latest_logid: 2,155    - threebot_name: me    - app_name: init    - latest_logid: 3,587]

JSX>