Issue with minemeld-sync.py and large lists

[adambaumeister]

Describe the bug

When using minemeld-sync.py, text lists with length > 10000 fail to upload with a HTTP 413 (request loo large) error code.

Expected behavior

Minemeld-sync should either slice the list and upload in multiple parts or the core engine should be updated to handle a larger request size.

Current behavior

HTTP 413 Payload Too Large

Possible solution

Slice the result data up and upload multipart

        results = result.values()
        json.dump(results,open("minemeld-dump.json", "w"))

        max = 10000
        if len(results) > max:
            ci = 0
            mi = len(results)
            while ci < mi:
                pi = ci
                ci = ci + max
                result_slice = results[pi:ci]
                print("Upload slice {}-{}/{}".format(pi, ci, mi))
                MM.upload(args.miner, '[{}]'.format(','.join(result_slice)))
        else:
                MM.upload(args.miner, '[{}]'.format(','.join(result.values())))

Steps to reproduce

1. Obtain a text list of domains/ips greater than 10k entries in length
2. run

   python minemeld-sync.py [params]

   Context

   Customer wants to use a URL blacklist that has approx 50k entries.

Your Environment

Minemeld running in Docker on RHEL.