Multiple errors when connecting without a proxy

[0xroot-bf]

Hey there,

I'm opening this ticket as I've observed someone had a similar issue in the past but the ticket was already closed.

I've decided to run your utility through a tmux session, and to my surprise I can't get ride off these these WSS errors:

python3 CertStreamMonitor.py -c test.conf
Looking for these strings: (.+\.)?domain1\.com$|(.+\.)?domain2\.com$|(.+\.)?domain3\.com$, detection threshold: 1
Connection established to CertStream! Listening for events...
Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Connection established to CertStream! Listening for events...

It happens every minute or two, usually takes longer if I'm not running the tool through tmux.

My installation was through pip3, after cloning the repo, and the conf file I'm using is the following one:

##
## Configuration file for CertStreamMonitor
##

[SEARCH]
# Keywords to look for (with '|' (or) as separator)
SearchKeywords = (.+\.)?domain1\.com$|(.+\.)?domain2\.com$|(.+\.)?domain3\.com$

# Minimum number of SearchKeywords detected in a hostname before writing it to DB. If you choose to set it to 2,
# the tool needs to detect 2 SearchKeywords matchs to write the match hostname to DB.
# Under 2 matching SearchKeywords but above 0, the matching hostname will only be logged to the log file.
DetectionThreshold = 1

# Keywords to ignore matched hosts (with '|' (or) as separator).
# Leave it empty or comment it to disable this feature.
# BlacklistKeywords = mail\.|\.com|bitcoin

[DATABASE]
# SQLite3 database path (will be created if not exist)
DBFile = ./db/CertStreamMonitor.sqlite3
TABLEname = CertMon

[CONNECT]
# (optional) Declare a HTTP or SOCKS proxy to use for HTTPS Get informations with scanhost.py
# ex: Proxy = http://127.0.0.1:8080 for a HTTP proxy server
# ex: Proxy = socks5://127.0.0.1:9050 for a SOCKS5 proxy server
#Proxy = socks5://127.0.0.1:9050

# Default user-agent (don't remove):
http_UA = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36
# Use a HTTPS user-agents file for scanhost.py
UAfile = ./useragent_list.txt

[LOGGING]
# Logging file (will be created if not exist)
LogFile = ./log/certstreammonitor.log

[REPORTING]
# Alerts reporting directory for scanhost.py
# %%m -> month
# %%d -> day
# %%Y -> year
# %%H -> hour
# %%M -> minute
Alerts_dir = ./alerts/%%Y/%%m/%%d

# Number of days scanhost.py will try to get information about an entry,
# after that the entry will not be checked anymore (default: 180 days).
Alert_Monitor_timelapse = 180

# (optional) Notifications: you can specify a notification destination. It will push to it each hostname detected as UP (+ all informations present in the JSON file).
# Notification Destination syntax and supported services are described on the apprise package Github page: https://github.com/caronc/apprise
# Example of Notification_Destination for sending email over SMTPS :
#Notification_Destination = mailtos://domain.xyz?user=smtp-user&pass=user-password&smtp=smtp-server-hostname&to=recipient-address&from=send-address&format=text&name=CertStreamMonitor-notification

[SERVER]
# Connection to Certficate Transparency aggregator server
ACTServer = wss://certstream.calidog.io

# (optional) Declare a HTTP proxy to use to connect to Certificate Transparency aggregator server
#Proxy_Host = 127.0.0.1
#Proxy_Port = 8080
#Proxy_Username = "Username"
#Proxy_Password = "My_Proxy-Password"

[SAFEBROWSING]
# (optional) Set the Google Safe Browsing API key value with your own if you want to test each hostname that is up against the GSB Lookup API
#Safe_Browsing_API_Key = value

I've attempted to run this as root, just to see if this could ring a bell, but still got the same issue.

Ultimately I've built my script on top of CertStream, and observed the same issue with the WSS connection. I'm running out of ideas, have you experienced this in the past?

Thanks,

Best

[0xroot-bf]

I did a bit more of research on this, and apparently is something not caused by CertStreamMonitor, but by the certstream-python wrapper.

I'm just curious if anyone else have experienced this issue. I know there are plenty tickets opened in the certstream-python repository, but I've attempted many things, including to install previous version of certstream (1.9) and upgrade websockets to 0.49, which didn't work. As for certstream 1.9 and websockets 0.48 i still experienced similar issue.

[t4d]

Hi @0xroot we observed this many times too, as I remember, the WebSocket behind CDN is regularly closed (RST). What we do, and it works better since that, we operate a CertStream-Server (https://github.com/CaliDog/CertStream-Server) docker instance near the CertStreamMonitor one. Hope it helps.

[0xroot-bf]

Hey @t4d thanks a lot for the suggestion, I will look into it!

[0xroot-bf]

Hey there, I thought it may be of your interest, but this issue has been resolved now @t4d :

https://github.com/CaliDog/certstream-python/issues/27

Cheers

[t4d]

Nice! thank you for the information!