search

deajan / traceroute_history

Keep traceroute histroy, make diffs. Use as standalone / smokeping companion
BSD 3-Clause "New" or "Revised" License
23 stars 5 forks source link

:: ERROR :: Cannot parse smokeping config file. #4

Open soymgomez opened 2 years ago

soymgomez commented 2 years ago

Hello @deajan ,

I am testing your project to use it together with our Smokeping.

When using the function to import Targets from Smokeping I always get this error:

2022-04-27 17:14:37,910 :: ERROR :: Cannot parse smokeping config file. We need as much titles as host entries.

my configuration line is as follows:

[SMOKEPING_SOURCE]
# Optional path to smokeping config file where hosts will be read
smokeping_config_path = /etc/smokeping/config.d/Targets

Can it be because of using @includes in our Targets file? My Targets file looks like this:

*** Targets ***

probe = FPing

menu = Top
title = Network Latency Grapher
remark = Welcome to the SmokePing website of <b>Insert Company Name Here</b>. \
         Here you will learn all about the latency of our network.

# 1. Group1
@include /etc/smokeping/config.d/group1.conf

# 2. Group2
@include /etc/smokeping/config.d/group2.conf

# 3. Group3
@include /etc/smokeping/config.d/group3.conf

...

Thank you very much

deajan commented 2 years ago

Indeed, I did a very basic implementation of smokeping target reading. I could add it quite quickly if you can send me a couple of example group files perhaps.

soymgomez commented 2 years ago

What we do is to split the Targets file into chunks.

Here is an example of the files from one of Smokeping's servers:

probe = FPing

menu = Top title = Network Latency Grapher remark = Welcome to the SmokePing website of Insert Company Name Here. \ Here you will learn all about the latency of our network.

1. ANYCAST

@include /etc/smokeping/config.d/anycast.conf

2. ISPs

@include /etc/smokeping/config.d/isps.conf

LibreNMS Targets

@include /etc/smokeping/config.d/librenms-targets.conf


 - /etc/smokeping/config.d/anycast.conf

++ AS60068 menu = ANYCAST CDN77 AS60068 title = ANYCAST CDN77 (AS60068 www.cdn77.com) probe = FPing host = www.cdn77.com

++ AS13335 menu = ANYCAST Cloudflare AS13335 title = ANYCAST Cloudflare (AS13335 1.1.1.1) probe = FPing host = 1.1.1.1

++ AS112 menu = ANYCAST DNS-OARC AS112 title = ANYCAST DNS-OARC (AS112 192.175.48.1) probe = FPing host = 192.175.48.1

++ AS15169-1 menu = ANYCAST Google API AS15169 title = ANYCAST Google API (AS15169 www.googleapis.com) probe = FPing host = www.googleapis.com

++ AS15169-2 menu = ANYCAST Google DNS AS15169 title = ANYCAST Google DNS (AS15169 8.8.8.8) probe = FPing host = 8.8.8.8

++ AS15169-3 menu = ANYCAST Google DRIVE AS15169 title = ANYCAST Google DRIVE (AS15169 drive.google.com) probe = FPing host = drive.google.com

++ AS19281 menu = ANYCAST Quad9 AS19281 title = ANYCAST Quad9 (AS19281 9.9.9.9) probe = FPing host = 9.9.9.9

++ AS2635 menu = ANYCAST WordPress AS2635 title = ANYCAST WordPress (AS2635 ns1.wordpress.com) probe = FPing host = ns1.wordpress.com

@include /etc/smokeping/config.d/anycast-rootDNS.conf


- /etc/smokeping/config.d/isps.conf

++ OVH menu = FR OVH AS16276 title = [FR] ISP OVH (AS16276 ovh.es) probe = FPing host = ovh.es

++ Online menu = FR Online.net AS12876 title = [FR] ISP Online.net (AS12876 nsa.online.net) probe = FPing host = nsa.online.net

++ Hetzner menu = DE Hetzner AS24940 title = [DE] ISP Hetzner (AS24940 hetzner.com) probe = FPing host = hetzner.com

deajan commented 2 years ago

Mind if I use those as files for tests (published on this git ?)

soymgomez commented 2 years ago

No problem, they are all public hosts! ;)

deajan commented 2 years ago

I pushed a proper smokeping config reader with tests, which passes the tests including the files you posted.

Be aware that I also moved everything into a traceroute_history subdirectory in order to make the directory scheme pip compatible, so you'll probably need to reconfigure paths if you are using git pull.

deajan commented 2 years ago

You should also pip install -r requirements.txt since I removed inplace requirements in favor of packages.

soymgomez commented 2 years ago

I'll try it out this afternoon/evening! Thank you very much

georgetasioulis commented 2 years ago

I downloaded the latest commit and tried to do a init-db but I'm getting the following error:

# ./traceroute_history_runner.py --config=traceroute_history.conf --init-db
Traceback (most recent call last):
  File "./traceroute_history_runner.py", line 32, in <module>
    from traceroute_history import config_management, trparse, schemas, models, crud
ModuleNotFoundError: No module named 'traceroute_history'
deajan commented 2 years ago

I needed to change the paths for it to become a pip package later. Until now, the easiest way for you to run it is to declare the path to the package, eg with the new path handing you would have:

/somepath/traceroute_history/traceroute_history/traceroute_history_runner.py

You would start to declare the path to the package before running with:

export PYTHONPATH=$PYTHONPATH:/somepath/traceroute_history

Then you can run like always

python3 /somepath/traceroute_history/traceroute_history/traceroute_history_runner.py [options]
soymgomez commented 2 years ago

I was testing it now on one of my servers and I got this error:

/opt/traceroute_history/traceroute_history# python3 traceroute_history_runner.py --config=traceroute_history.conf --config=traceroute_history.conf --update-now

2022-05-05 22:46:59,005 :: INFO :: Scheduler started
Traceback (most recent call last):
  File "/opt/traceroute_history/traceroute_history/traceroute_history_runner.py", line 577, in <module>
    main(sys.argv[1:])
  File "/opt/traceroute_history/traceroute_history/traceroute_history_runner.py", line 568, in main
    execute()
  File "/opt/traceroute_history/traceroute_history/traceroute_history_runner.py", line 439, in execute
    grp = [schemas.GroupCreate(name=grp_name) for grp_name in config_management.get_groups_from_config(config, target_name)]
  File "/opt/traceroute_history/traceroute_history/config_management.py", line 189, in get_groups_from_config
    return [group.strip() for group in config['TARGET:' + target]['groups'].split(',')]
TypeError: can only concatenate str (not "dict") to str

In case it helps, I paste here my entire configuration file. The configuration is all default except for the Smokeping configuration, the path to the SQLlite file and the example Targets are commented out.

[TRACEROUTE_HISTORY]

# Config file revision 0.5.3 2020092202

# Install dir from which everythying is variable
#install_dir = /opt/traceroute_history
install_dir = C:\GIT\traceroute_history

# Database driver, as of v0.5.1, only sqlite / mysql are supported
database_driver = sqlite

# Might be an IP / URL
# When using sqlite, it will be the full path to the database file
database_host = /opt/traceroute_history/sqltest.db
#database_host = C:\GIT\traceroute_history\sqltest.db

# Optional database info (does not apply to sqlite driver)
database_name =
database_user =
database_password =

# Traceroute probe interval
interval = 1800

# Traceroute increased rtt detection threshold (ms)
rtt_detection_threshold = 50

# After how many days old traceroutes should be deleted
delete_history_days = 365

# Minimum number of traceroutes to keep anytime
minimum_keep = 10

# full path to log file path
log_file = /var/log/traceroute_history.log

[UI_SETTINGS]

bind_to = 0.0.0.0
bind_port = 5001

# When using a reverse proxy, please add the following directive (example for apache)
# ProxyPreserveHost On
# When using a SSL reverse proxy, set the following header (example for apache)
# RequestHeader set X-Forwarded-Proto "https"

# Optional reverse public served URL for reverse proxys
#sub_directory = /traceroute_history

[ALERT_SETTINGS]

alert_on_rtt_detection = yes
alert_on_changing_hops = yes

# List of groups that alerts work for (if not set, all hosts will be alerted)
only_alert_on_groups =

alert_type = mail
sender_mail =
destination_mails =
smtp_server =
smtp_port =
smtp_user =
smtp_password =

# accepts ssl, tls or none
smtp_security = none
subject = Traceroute History found issue on host {0}

[SMOKEPING_SOURCE]
# Optional path to smokeping config file where hosts will be read
smokeping_config_path = /etc/smokeping/config.d/Targets

[SMOKEPING_GROUP1]
# Optional smokeping groups (there may be an unlimited number of SMOKEPING_GROUP sections
#smokeping_group =

# Filter hostnames based on regex
exclude_hostname_regex =

# Include only hostnames that match the following regex
include_hostname_regex = '.*'

#### TARGET DECLARATION
# The following sections must be in form of [TARGET:name]

#[TARGET:some search engine]
# Name can be anything
# Address needs to be a FQDN, an IPv4 or IPv6
#address = google.com
# Comma separated list of groups the host belongs to
#groups = mygroup

#[TARGET:kernel.org]
#address = kernel.org
#groups = somegroup, mygroup

#[TARGET:nope]
#address = somenonfoundip.local
#groups = none
georgetasioulis commented 2 years ago

I needed to change the paths for it to become a pip package later. Until now, the easiest way for you to run it is to declare the path to the package, eg with the new path handing you would have:

[...]

Sorry, no difference:

# export PYTHONPATH=$PYTHONPATH:/opt/traceroute_history
# python3 /opt/traceroute_history/traceroute_history_runner.py --config=traceroute_history.conf --init-db
Traceback (most recent call last):
  File "/opt/traceroute_history/traceroute_history_runner.py", line 32, in <module>
    from traceroute_history import config_management, trparse, schemas, models, crud
ModuleNotFoundError: No module named 'traceroute_history'

My traceroute_history.conf is pretty much identical to @soymgomez 's with the only difference being the smokeping_config_path line where I don't have a path, but a single file to parse:

[SMOKEPING_SOURCE]
# Optional path to smokeping config file where hosts will be read
smokeping_config_path =/opt/smokeping/etc/config
deajan commented 2 years ago

Sorry, no difference:


# export PYTHONPATH=$PYTHONPATH:/opt/traceroute_history
# python3 /opt/traceroute_history/traceroute_history_runner.py --config=traceroute_history.conf --init-db

@georgetasioulis Please have a look at the path I stated. In your case, you would have export PYTHONPATH=$PYTHONPATH:/opt since you don't have a traceroute_history package subdirectory in the traceroute_history git directory.

deajan commented 2 years ago

@soymgomez Thanks for sharing your config file, makes things easier. I've pushed commit https://github.com/deajan/traceroute_history/commit/795ec575f2bea5f47415b19a2968bd7053a49b19 that fixes the targets from smokeping to look like those from normal config file. Can you give it a spin perhaps ?

soymgomez commented 2 years ago

@deajan I just tested it on my test server and it works fine.

I'm going to try to fix the integration with Nginx as I have to add the Path to avoid the error:

2022/05/06 16:01:09 [error] 1192589#1192589: *1 FastCGI sent in stderr: "Traceback (most recent call last):
  File "/usr/lib/cgi-bin/smokeping_th_helper.fcgi", line 22, in <module>
    from traceroute_history import traceroute_history_runner
ModuleNotFoundError: No module named 'traceroute_history'" while reading response header from upstream, client: xx.xx.xx.xx, server: smokeping.xxxxxxx.net, request: "GET /smokeping/smokeping_th_helper.fcgi?target=ANYCAST.AS60068 HTTP/2.0", upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: "smokeping.xxxxxxx.net"
deajan commented 2 years ago

Since you moved smokeping_th_helper.fgci from traceroute_history directory to /usr/lib/cgi-bin, you have to add that path as env for nginx, or directly into the script. In the fcgi script you could add the following

sys.path.append('/opt/traceroute_history')

Of course, using fastcgi_param PYTHONPATH /opt/traceroute_history seems an easier idea to me.