rhsimplex / image-match

🎇 Quickly search over billions of images
2.94k stars 405 forks source link

try to add_image with multiprocessing.Pool don't work #103

Open AmyWANGC opened 6 years ago

AmyWANGC commented 6 years ago

Here is my test code:

`

import multiprocessing import time import elasticsearch from image_match.elasticsearch_driver import SignatureES es = elasticsearch.Elasticsearch() es.indices.create('tester5') ses5 = SignatureES(es, index='tester5')

def func(i,line): try: ses5.add_image(line.strip()) print i return i except Exception,Argument: print "Exception: %s" % Argument return "Exception: %s" % Argument

if name == "main": pool = multiprocessing.Pool(processes=4) results = [] i=1 with open('/Users/amy/etest/ugcpic1540448179514.txt', 'r') as f: for line in f.readlines(): i=i+1 if i>0: results.append(pool.apply_async(func, (i,line, ))) pool.close() pool.join() print ("Sub-process(es) done.")
for res in results: print (res.get()) `

And then it turns out there is no exception and tester5 has no pics inside. image

rhsimplex commented 6 years ago

Could be something with pickling the DB connection. Try using a pool with an initializer and create the ses5 object within the spawned process.