Push hangs (sometimes) with no output

[ghost]

Upon running the following the push section of the code hangs for some time. Sometimes never completing.

# python AuthNotif.py
Checking authorisation status for 1 users
Sending a notification to Ben Bridges

Code used:

#!/usr/bin/python3

#? Import required modules
import PushNotif
import mysql.connector
import json
import random

#? Create Dict to hold push notification info
DictData ={}

Greetings=["Hi, ","Hello, ","How are you, ", "I come in peace, ", "Ahoy, ", "Greetings, "]

#? Group of Different functions for different styles
class style():
    BLACK = lambda x: '\033[30m' + str(x)
    RED = lambda x: '\033[31m' + str(x)
    GREEN = lambda x: '\033[32m' + str(x)
    YELLOW = lambda x: '\033[33m' + str(x)
    BLUE = lambda x: '\033[34m' + str(x)
    MAGENTA = lambda x: '\033[35m' + str(x)
    CYAN = lambda x: '\033[36m' + str(x)
    WHITE = lambda x: '\033[37m' + str(x)
    UNDERLINE = lambda x: '\033[4m' + str(x)
    RESET = lambda x: '\033[0m' + str(x)

#? Create conection to mySQL DB
mydb = mysql.connector.connect(
  host="localhost",
  user="XXX",
  unix_socket="/var/run/mysqld/mysqld.sock",
  passwd="XXX",
  database="XXX"
)

#? Create DB cursor. Will return dict
mycursor = mydb.cursor(dictionary=True)

#? Execute query to get a list of authorisors with push notifications turned on
mycursor.execute("SELECT username, fullname, push, department FROM accounts WHERE admin=1 AND push IS NOT NULL")

#? Get all the results
myresult = mycursor.fetchall()

print(style.YELLOW("Checking authorisation status for "+ str(len(myresult)) +" users"))

#? Loop through results
for user in myresult:
  #? Get no of time sheets to auth in users dept
  mycursor.execute("SELECT COUNT(DISTINCT(EmpNum)) AS 'Users', COUNT(*) AS 'NoToAuth' FROM timesheet WHERE Auth IS NULL AND Department='"+user['department']+"'")
  #? Get query result
  results = mycursor.fetchone()
  #? If there are more than 0 sheets to auth then we need to send a notification
  if (results['NoToAuth']>0):
    #? Load the users push info
    push_info=json.loads(user['push'])
    #? Get the users first name only
    firstname = user['fullname'].split(None, 1)[0]

    #? Decide if plural or singlar context is needed in message
    if(results['NoToAuth']==1 and results['Users']==1):
      DictData['message']=random.choice(Greetings)+firstname+"! You have 1 job to authorise today from 1 employee"
    elif(results['NoToAuth']>1 and results['Users']==1):
      DictData['message']=random.choice(Greetings)+firstname+"! You have "+str(results['NoToAuth'])+" jobs to authorise today from 1 employee"
    else:
      DictData['message']=random.choice(Greetings)+firstname+"! You have "+str(results['NoToAuth'])+" jobs to authorise today from "+str(results['Users'])+" employees"

    #? Set the notification title
    DictData['title']="Authorise Your Time Sheets"

    #? Set the notification tag
    DictData['tag']="Authorise"

    #? Renotify on new notification with the same tag
    DictData['renotify']="true"

    #? Send the notification
    print("Sending a notification to "+user['fullname'])
    try:
      PushNotif.send(json.dumps(DictData),"XXX",push_info)
    except Exception as e:
      print(e)
      continue
  else:
    print("Nothing to authorise for "+user['fullname'])

PushNotif:

#!/usr/bin/python3

#? Import required module for web push notifications
from pywebpush import webpush, WebPushException

#? Send sub. Requires Data in json string, Private key as string and push info as json string
def send(Data, PrivateKey, PushInfo):
    #? Try webpush with provided info
    try:
        webpush(
        subscription_info=PushInfo,
        data=Data,
        vapid_private_key=PrivateKey,
        vapid_claims={"sub": "mailto:YourNameHere@example.org",}
        )
    #? Handle push fail
    except WebPushException as ex:
        return("I'm sorry, Dave, but I can't do that: {}", repr(ex))
        #! Mozilla returns additional information in the body of the response.
        if ex.response and ex.response.json():
            extra = ex.response.json()
            return("Remote service replied with a {}:{}, {}",extra.code,extra.errno,extra.message)

As you can see from the print statements, it seems to hang at the point of trying to send the notification. Sometimes it hangs, but others it fires straight away. Is there anyway I can get more output from the webpush command to see whats occurring?

[ghost]

Ok i have some more info for you. I know exactly where it hangs now. Here is the output from gdb:

(gdb) py-bt
Traceback (most recent call first):
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 146, in _get_osurandom_engine
    res = self._lib.ENGINE_init(e)
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 163, in activate_osrandom_engine
    with self._get_osurandom_engine() as e:
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 119, in __init__
    self.activate_osrandom_engine()
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 2419, in <module>
    backend = Backend()
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/usr/local/lib/python2.7/dist-packages/py_vapid/__init__.py", line 65, in from_raw
    backend=default_backend())
  File "/usr/local/lib/python2.7/dist-packages/py_vapid/__init__.py", line 142, in from_string
    return cls.from_raw(pkey)
  File "/usr/local/lib/python2.7/dist-packages/pywebpush/__init__.py", line 415, in webpush
    vv = Vapid.from_string(private_key=vapid_private_key)
  File "/root/python-scripts/PushNotif.py", line 14, in send
    vapid_claims={"sub": "mailto:YourNameHere@example.org",}
  File "AuthNotif.py", line 68, in <module>
(gdb) py-list
 141            # Fetches an engine by id and returns it. This creates a structural
 142            # reference.
 143            e = self._lib.ENGINE_by_id(self._lib.Cryptography_osrandom_engine_id)
 144            self.openssl_assert(e != self._ffi.NULL)
 145            # Initialize the engine for use. This adds a functional reference.
>146            res = self._lib.ENGINE_init(e)
 147            self.openssl_assert(res == 1)
 148
 149            try:
 150                yield e
 151            finally:

So we can see it is having issues with res = self._lib.ENGINE_init(e) Values of e and self when it hangs:

self = <Backend(_ffi=<CompiledFFI at remote 0x7fe102ef8050>, _lib=<module at remote 0x7fe102ef9c20>, _cipher_registry={(<type at remote 0x55a3fb0032e0>, <type at remote 0x55a3fb007710>): <GetCipherByName(_fmt='des-ede3') at remote 0x7fe1028f4c50>, (<type at remote 0x55a3faff6a80>, <type at remote 0x55a3fb0089f0>): <GetCipherByName(_fmt='seed-{mode.name}') at remote 0x7fe102889290>, (<type at remote 0x55a3fafbba90>, <type at remote 0x55a3fb006970>): <GetCipherByName(_fmt='{cipher.name}-{cipher.key_size}-{mode.name}') at remote 0x7fe1028f4c90>, (<type at remote 0x55a3fb0032e0>, <type at remote 0x55a3fb008db0>): <GetCipherByName(_fmt='des-ede3-{mode.name}') at remote 0x7fe1028f4fd0>, (<type at remote 0x55a3faff6a80>, <type at remote 0x55a3fb007710>): <GetCipherByName(_fmt='seed-{mode.name}') at remote 0x7fe1028892d0>, (<type at remote 0x55a3faff6530>, <type at remote 0x55a3fb0089f0>): <GetCipherByName(_fmt='{cipher.name}-{mode.name}') at remote 0x7fe102889410>, (<type at remote 0x55a3faff71e0>, <type at remote 0x55a3f9...(truncated)

e = <_cffi_backend.CData at remote 0x7fe102ec55d0>

I am completely lost with all this info but hopefully it's enough for you to help me out.

[ghost]

It's just so strange how sometimes it will work fine, and other times it will hang. I'm not sure if this is relevant, but I'm running this on a Digital Ocean Droplet running Ubuntu 18.04.2. The issue happens with python 2.7.15rc1 and python 3.6.7

[ghost]

Well it seems to be an issue that only occurs on my Digital Ocean Droplet. I just ran the same code about 30 times in a row on my Windows 10 machine and it worked perfect every time.

[jrconlin]

Well, first, thank you for the added info. I see that the problem is inside openssl when trying to get a proper random number from the OS. That's what

File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 163, in activate_osrandom_engine
    with self._get_osurandom_engine() as e:

is doing.

I've seen similar sorts of things happen on systems with low entropy. Basically, there's not enough sources of truly random crap happening that a proper random number could be derived. If this is on a shared system, it's even worse because entropy tends to be shared among all containers. If you're on a box with some bitcoin miner or doing a bunch of TLS cert management, then yeah, you're not going to have a lot of entropy to draw from.

(That also might explain why your home box has no problem.)

I'm not really sure what to suggest to make this better. I suppose you could wrap the call with a timeout and retry when things are a bit less busy. I'd STRONGLY suggest you don't try to use something other than your OS urandom, so using a PRNG is probably not-ideal for anything close to production.

Sorry I can't offer more help.

[ghost]

That's no problem. Thanks for the info. At least I know what the issue is now and I can investigate a solution.

[ghost]

For anyone else who stumbles across this: https://www.digitalocean.com/community/tutorials/how-to-setup-additional-entropy-for-cloud-servers-using-haveged