create sqs broker

[komuw]

wiji itself should not depend on any third-party libraries. Since the sqs broker will have to probably depend on some third party libs, eg boto3, then the sqs broker should not be part of the wiji python lib.

The sqs broker however can share the same repo as wiji: they just have to be two independent python libs with their own setup.py files.

The sqs broker should ideally, use an asyncio AWS library.
I do not think boto3 is asyncio as yet. As such;

• we may have to use another one
• or run boto3 using concurrent.futures.ThreadPoolExecutor

[komuw]

• we should add compression to reduce payload size and hence reduce cost of SQS
• Queue and deque 1mil tasks on SQS for every merge to master
• We should also offer an option to batch upto 10 requests(note that SQS has a limit of about 256KB)

[komuw]

use, sys.getsizeof;

import json, sys
du =json.dumps(x)
len(du) #653
sys.getsizeof(du) #702 (bytes) <- this is actual size

import zlib

ccc=zlib.compress(du.encode())
len(ccc) #141
sys.getsizeof(ccc) #174B

json.loads(zlib.decompress(ccc)) # returns dict