When using minemeld-sync.py, text lists with length > 10000 fail to upload with a HTTP 413 (request loo large) error code.
Expected behavior
Minemeld-sync should either slice the list and upload in multiple parts or the core engine should be updated to handle a larger request size.
Current behavior
HTTP 413 Payload Too Large
Possible solution
Slice the result data up and upload multipart
results = result.values()
json.dump(results,open("minemeld-dump.json", "w"))
max = 10000
if len(results) > max:
ci = 0
mi = len(results)
while ci < mi:
pi = ci
ci = ci + max
result_slice = results[pi:ci]
print("Upload slice {}-{}/{}".format(pi, ci, mi))
MM.upload(args.miner, '[{}]'.format(','.join(result_slice)))
else:
MM.upload(args.miner, '[{}]'.format(','.join(result.values())))
Steps to reproduce
Obtain a text list of domains/ips greater than 10k entries in length
run
python minemeld-sync.py [params]
Context
Customer wants to use a URL blacklist that has approx 50k entries.
Describe the bug
When using minemeld-sync.py, text lists with length > 10000 fail to upload with a HTTP 413 (request loo large) error code.
Expected behavior
Minemeld-sync should either slice the list and upload in multiple parts or the core engine should be updated to handle a larger request size.
Current behavior
HTTP 413 Payload Too Large
Possible solution
Slice the result data up and upload multipart
Steps to reproduce
Context
Customer wants to use a URL blacklist that has approx 50k entries.
Your Environment
Minemeld running in Docker on RHEL.