search

jina-ai / jina

☁️ Build multimodal AI applications with cloud-native stack
https://docs.jina.ai
Apache License 2.0
20.98k stars 2.22k forks source link

ContainerPea fails to stop when replicas > 1 #262

Closed nan-wang closed 4 years ago

nan-wang commented 4 years ago

During indexing, ContainerPea with replicas > 1 fails to stop after indexing completed.

nan-wang commented 4 years ago

check out the codes at jina-ai/example/webqa-search-with-container.

run codes with

python index.py

Following are the logs

/Users/nanwang/.pyenv/versions/3.7.5/bin/python3.7 /Users/nanwang/Codes/jina-ai/examples/webqa-search/index.py
   BaseExecutor@1444[I]:post initiating, this may take some time...
   BaseExecutor@1444[I]:post initiating, this may take some time takes 0.000 secs
     GatewayPea@1444[S]:gateway is listening at: 0.0.0.0:63856
WebQATitleExtra@1447[I]:post initiating, this may take some time...
WebQATitleExtra@1447[I]:post initiating, this may take some time takes 0.000 secs
WebQATitleExtra@1447[S]:successfully built WebQATitleExtractor from a yaml config
title_extractor@1447[I]:setting up sockets...
title_extractor@1447[I]:input tcp://0.0.0.0:63854 (SUB_CONNECT)      output tcp://0.0.0.0:63865 (PUSH_CONNECT)   control over tcp://0.0.0.0:63859 (PAIR_BIND)
title_extractor@1447[S]:ready and listening
 LeveldbIndexer@1449[I]:post initiating, this may take some time...
 LeveldbIndexer@1449[I]:post initiating, this may take some time takes 0.000 secs
 LeveldbIndexer@1449[S]:restore LeveldbIndexer from /tmp/jina/webqa/title_meta_doc_indexer.bin
title_meta_doc_@1449[I]:setting up sockets...
title_meta_doc_@1449[I]:input tcp://0.0.0.0:63854 (SUB_CONNECT)      output tcp://0.0.0.0:63879 (PUSH_CONNECT)   control over tcp://0.0.0.0:63863 (PAIR_BIND)
title_meta_doc_@1449[S]:ready and listening
   BaseExecutor@1450[I]:post initiating, this may take some time...
   BaseExecutor@1450[I]:post initiating, this may take some time takes 0.000 secs
   BaseExecutor@1450[S]:successfully built BaseExecutor from a yaml config
title_encoder-h@1450[I]:setting up sockets...
title_encoder-h@1450[I]:input tcp://0.0.0.0:63865 (PULL_BIND)    output tcp://0.0.0.0:63870 (ROUTER_BIND)    control over tcp://0.0.0.0:63869 (PAIR_BIND)
title_encoder-h@1450[S]:ready and listening
   BaseExecutor@1451[I]:post initiating, this may take some time...
   BaseExecutor@1451[I]:post initiating, this may take some time takes 0.001 secs
   BaseExecutor@1451[S]:successfully built BaseExecutor from a yaml config
title_encoder-t@1451[I]:setting up sockets...
title_encoder-t@1451[I]:input tcp://0.0.0.0:63871 (PULL_BIND)    output tcp://host.docker.internal:63875 (PUSH_CONNECT)  control over tcp://0.0.0.0:63872 (PAIR_BIND)
title_encoder-t@1451[S]:ready and listening
title_encoder-1@1452[I]:waiting ready signal from the container
🐳 JINA@ 1[I]:
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMNNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMWxxxxxxxxxOMMMMMNxxxxxxxxx0MMMMMKddddddxkKWMMMMMMMMMMMMXOxdddxONMMMM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllo0MMMMMMMM0olllllllllo0MM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOlllllllllllloWMMMMMdllllllllllllldM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllllloMMMM0lllllllllllllllK
🐳 MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllllKMMM0lllllllllllllllO
🐳 MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMollllllllllllllO
🐳 MWOkkkkk0MMMKlllllllllkMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMxlllllllllllllO
🐳 NkkkkkkkkkMMKlllllllloMMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMMWOdolllllllllO
🐳 KkkkkkkkkkNMKllllllldMMMMMMMMWWWWWWWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MOkkkkkkk0MMKllllldXMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMWX00KXMMMMXxk0XMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 
🐳 ▶️  /opt/conda/bin/jina pod --yaml-path roberta.yml --name title_encoder --identity f23184ba54 --port-in 63870 --port-out 63871 --host-in host.docker.internal --host-out host.docker.internal --socket-in DEALER_CONNECT --socket-out PUSH_CONNECT --port-ctrl 63873 --timeout-ready 1000000 --replica-id 1 --port-grpc 63868
🐳 array-in-pb = False
🐳 check-version = False
🐳 cli = pod
🐳 ctrl-with-ipc = False
🐳 dump-interval = 240
🐳 entrypoint = None
🐳 exit-no-dump = False
🐳 host = 0.0.0.0
🐳 🔧️                        host-in = host.docker.internal          
🐳 🔧️                       host-out = host.docker.internal          
🐳 🔧️                       identity = f23184ba54                    
🐳 image = None
🐳 log-profile = False
🐳 log-remote = False
🐳 log-sse = False
🐳 logserver = False
🐳 logserver-config = /opt/conda/lib/python3.7/site-
🐳 max-idle-time = 60
🐳 max-message-size = -1
🐳 memory-hwm = -1
🐳 🔧️                           name = title_encoder                 
🐳 num-part = 1
🐳 override-exec-log = False
🐳 polling = ANY
🐳 🔧️                      port-ctrl = 63873                         
🐳 🔧️                      port-grpc = 63868                         
🐳 🔧️                        port-in = 63870                         
🐳 🔧️                       port-out = 63871                         
🐳 proxy = False
🐳 pull-latest = False
🐳 py-modules = None
🐳 read-only = False
🐳 reducing-yaml-path = _forward
🐳 🔧️                     replica-id = 1                             
🐳 replicas = 1
🐳 runtime = process
🐳 scheduling = LOAD_BALANCE
🐳 separated-workspace = False
🐳 shutdown-idle = False
🐳 🔧️                      socket-in = DEALER_CONNECT                
🐳 🔧️                     socket-out = PUSH_CONNECT                  
🐳 timeout = -1
🐳 timeout-ctrl = 5000
🐳 🔧️                  timeout-ready = 1000000                       
🐳 volumes = None
🐳 🔧️                      yaml-path = roberta.yml                   
🐳 
🐳 TransformerRobe@ 8[I]:post initiating, this may take some time...
🐳 TransformerRobe@ 8[I]:post initiating, this may take some time takes 9.509 secs
🐳 TransformerRobe@ 8[S]:successfully built TransformerRobertaEncoder from a yaml config
🐳 title_encoder@ 8[I]:setting up sockets...
🐳 title_encoder@ 8[I]:input tcp://host.docker.internal:63870 (DEALER_CONNECT)    output tcp://host.docker.internal:63871 (PUSH_CONNECT)  control over tcp://0.0.0.0:63873 (PAIR_BIND)
title_encoder-h@1450[I]:received "control" from title_encoder▸⚐
title_encoder-1@1452[S]:ready and listening
🐳 title_encoder@ 8[S]:ready and listening
title_encoder-2@1454[I]:waiting ready signal from the container
🐳 JINA@ 1[I]:
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMNNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMWxxxxxxxxxOMMMMMNxxxxxxxxx0MMMMMKddddddxkKWMMMMMMMMMMMMXOxdddxONMMMM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllo0MMMMMMMM0olllllllllo0MM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOlllllllllllloWMMMMMdllllllllllllldM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllllloMMMM0lllllllllllllllK
🐳 MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllllKMMM0lllllllllllllllO
🐳 MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMollllllllllllllO
🐳 MWOkkkkk0MMMKlllllllllkMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMxlllllllllllllO
🐳 NkkkkkkkkkMMKlllllllloMMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMMWOdolllllllllO
🐳 KkkkkkkkkkNMKllllllldMMMMMMMMWWWWWWWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MOkkkkkkk0MMKllllldXMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMWX00KXMMMMXxk0XMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 
🐳 ▶️  /opt/conda/bin/jina pod --yaml-path roberta.yml --name title_encoder --identity 0609d26720 --port-in 63870 --port-out 63871 --host-in host.docker.internal --host-out host.docker.internal --socket-in DEALER_CONNECT --socket-out PUSH_CONNECT --port-ctrl 63874 --timeout-ready 1000000 --replica-id 2 --port-grpc 63868
🐳 array-in-pb = False
🐳 check-version = False
🐳 cli = pod
🐳 ctrl-with-ipc = False
🐳 dump-interval = 240
🐳 entrypoint = None
🐳 exit-no-dump = False
🐳 host = 0.0.0.0
🐳 🔧️                        host-in = host.docker.internal          
🐳 🔧️                       host-out = host.docker.internal          
🐳 🔧️                       identity = 0609d26720                    
🐳 image = None
🐳 log-profile = False
🐳 log-remote = False
🐳 log-sse = False
🐳 logserver = False
🐳 logserver-config = /opt/conda/lib/python3.7/site-
🐳 max-idle-time = 60
🐳 max-message-size = -1
🐳 memory-hwm = -1
🐳 🔧️                           name = title_encoder                 
🐳 num-part = 1
🐳 override-exec-log = False
🐳 polling = ANY
🐳 🔧️                      port-ctrl = 63874                         
🐳 🔧️                      port-grpc = 63868                         
🐳 🔧️                        port-in = 63870                         
🐳 🔧️                       port-out = 63871                         
🐳 proxy = False
🐳 pull-latest = False
🐳 py-modules = None
🐳 read-only = False
🐳 reducing-yaml-path = _forward
🐳 🔧️                     replica-id = 2                             
🐳 replicas = 1
🐳 runtime = process
🐳 scheduling = LOAD_BALANCE
🐳 separated-workspace = False
🐳 shutdown-idle = False
🐳 🔧️                      socket-in = DEALER_CONNECT                
🐳 🔧️                     socket-out = PUSH_CONNECT                  
🐳 timeout = -1
🐳 timeout-ctrl = 5000
🐳 🔧️                  timeout-ready = 1000000                       
🐳 volumes = None
🐳 🔧️                      yaml-path = roberta.yml                   
🐳 
🐳 TransformerRobe@ 8[I]:post initiating, this may take some time...
🐳 TransformerRobe@ 8[I]:post initiating, this may take some time takes 31.922 secs
🐳 TransformerRobe@ 8[S]:successfully built TransformerRobertaEncoder from a yaml config
🐳 title_encoder@ 8[I]:setting up sockets...
🐳 title_encoder@ 8[I]:input tcp://host.docker.internal:63870 (DEALER_CONNECT)    output tcp://host.docker.internal:63871 (PUSH_CONNECT)  control over tcp://0.0.0.0:63874 (PAIR_BIND)
title_encoder-h@1450[I]:received "control" from title_encoder▸⚐
title_encoder-2@1454[S]:ready and listening
🐳 title_encoder@ 8[S]:ready and listening
   NumpyIndexer@1464[I]:post initiating, this may take some time...
   NumpyIndexer@1464[I]:post initiating, this may take some time takes 0.000 secs
   NumpyIndexer@1464[S]:restore NumpyIndexer from /tmp/jina/webqa/title_vecidx_exec.bin
 LeveldbIndexer@1464[I]:post initiating, this may take some time...
 LeveldbIndexer@1464[I]:post initiating, this may take some time takes 0.000 secs
 LeveldbIndexer@1464[S]:restore LeveldbIndexer from /tmp/jina/webqa/title_chunk_meta_exec.bin
CompoundExecuto@1464[I]:post initiating, this may take some time...
CompoundExecuto@1464[I]:post initiating, this may take some time takes 0.000 secs
CompoundExecuto@1464[S]:successfully built CompoundExecutor from a yaml config
title_compound_@1464[I]:setting up sockets...
title_compound_@1464[I]:input tcp://0.0.0.0:63875 (PULL_BIND)    output tcp://0.0.0.0:63879 (PUSH_CONNECT)   control over tcp://0.0.0.0:63877 (PAIR_BIND)
title_compound_@1464[S]:ready and listening
   BaseExecutor@1465[I]:post initiating, this may take some time...
   BaseExecutor@1465[I]:post initiating, this may take some time takes 0.000 secs
   BaseExecutor@1465[S]:successfully built BaseExecutor from a yaml config
         joiner@1465[I]:setting up sockets...
         joiner@1465[I]:input tcp://0.0.0.0:63879 (PULL_BIND)    output tcp://0.0.0.0:63880 (PUSH_BIND)  control over tcp://0.0.0.0:63881 (PAIR_BIND)
         joiner@1465[S]:ready and listening
           Flow@1444[I]:6 Pods (i.e. 9 Peas) are running in this Flow
           Flow@1444[S]:flow is now ready for use, current build_level is GRAPH
           JINA@1444[W]:parser Client can not recognize the following args: ['--prefetch', '10'], they are ignored
       PyClient@1444[S]:connected to the gateway at 0.0.0.0:63856!
index [=                   ] 📃      0 ⏱️ 0.0s 🐎 0.0/s      0 batchindex ...         gateway@1444[I]:setting up sockets...
        gateway@1444[I]:input tcp://0.0.0.0:63880 (PULL_CONNECT)     output tcp://0.0.0.0:63854 (PUB_BIND)   control over ipc:///var/folders/6z/fwt6w1tx56nfk69ry1p265zc0000gp/T/tmp51fe8b1a (PAIR_BIND)
        gateway@1444[I]:prefetching 50 requests...
        gateway@1444[W]:if this takes too long, you may want to take smaller "--prefetch" or ask client to reduce "--batch-size"
total docs: 55945
        gateway@1444[I]:prefetching 50 requests takes 1.258 secs
        gateway@1444[I]:send: 0 recv: 0 pending: 0
title_extractor@1447[I]:received "index" from gateway▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_extractor@1447[I]:received "index" from gateway▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
title_extractor@1447[I]:received "index" from gateway▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
title_meta_doc_@1449[I]:received "index" from gateway▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
         joiner@1465[I]:received "index" from gateway▸title_meta_doc_indexer▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "index" from gateway▸title_extractor▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
🐳 title_encoder@ 8[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸⚐
🐳 TransformerRobe@ 8[I]:batching enabled for TransformerRobertaEncoder.encode(). batch_size=8   num_batch=None  axis=0
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-h@1450[I]:received "control" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
title_encoder-t@1451[I]:received "index" from gateway▸title_extractor▸title_encoder-head▸title_encoder▸⚐
    [81.857 secs]
    ✅ done in ⏱ 81.9s 🐎 0.0/s
title_meta_doc_@1449[W]:user cancel the process
title_extractor@1447[W]:user cancel the process
title_encoder-t@1451[W]:user cancel the process
title_compound_@1464[W]:user cancel the process
title_encoder-h@1450[W]:user cancel the process
         joiner@1465[W]:user cancel the process
title_encoder-2@1454[W]:user cancel the process
       PyClient@1444[W]:user cancel the process
title_encoder-1@1452[W]:user cancel the process
   BaseExecutor@1465[I]:no update since 2020-04-17 10:40:39, will not save. If you really want to save it, call "touch()" before "save()" to force saving
   NumpyIndexer@1464[I]:no update since 2020-04-17 10:30:49, will not save. If you really want to save it, call "touch()" before "save()" to force saving
   BaseExecutor@1450[I]:no update since 2020-04-17 10:39:53, will not save. If you really want to save it, call "touch()" before "save()" to force saving
   BaseExecutor@1451[I]:no update since 2020-04-17 10:39:53, will not save. If you really want to save it, call "touch()" before "save()" to force saving
         joiner@1465[I]:executor says there is nothing to save
title_encoder-h@1450[I]:executor says there is nothing to save
WebQATitleExtra@1447[I]:no update since 2020-04-17 10:39:53, will not save. If you really want to save it, call "touch()" before "save()" to force saving
title_encoder-t@1451[I]:executor says there is nothing to save
title_extractor@1447[I]:executor says there is nothing to save
 LeveldbIndexer@1464[I]:no update since 2020-04-17 10:30:49, will not save. If you really want to save it, call "touch()" before "save()" to force saving
CompoundExecuto@1464[I]:no update since 2020-04-17 10:40:39, will not save. If you really want to save it, call "touch()" before "save()" to force saving
 LeveldbIndexer@1449[S]:artifacts of this executor (title_meta_doc_indexer) is persisted to /tmp/jina/webqa/title_meta_doc_indexer.bin
title_compound_@1464[I]:dumped changes to the executor,  82s since last the save
       PyClient@1444[S]:terminated
title_meta_doc_@1449[I]:dumped changes to the executor, 128s since last the save
title_encoder-t@1451[I]:msg_sent: 13 bytes_sent: 5 KB msg_recv: 13 bytes_recv:4 KB
title_extractor@1447[I]:msg_sent: 13 bytes_sent: 5 KB msg_recv: 13 bytes_recv:3 KB
title_encoder-h@1450[I]:msg_sent: 13 bytes_sent: 5 KB msg_recv: 28 bytes_recv:7 KB
title_extractor@1447[S]:terminated
title_encoder-h@1450[S]:terminated
title_encoder-t@1451[S]:terminated
         joiner@1465[I]:msg_sent: 0 bytes_sent: 0 KB msg_recv: 13 bytes_recv:3 KB
         joiner@1465[S]:terminated
title_meta_doc_@1449[I]:msg_sent: 13 bytes_sent: 3 KB msg_recv: 13 bytes_recv:3 KB
title_meta_doc_@1449[S]:terminated
title_compound_@1464[I]:msg_sent: 0 bytes_sent: 0 KB msg_recv: 0 bytes_recv:0 KB
title_compound_@1464[S]:terminated
     GatewayPea@1444[S]:terminated
           Flow@1444[S]:flow is closed and all resources should be released already, current build level is EMPTY

Process finished with exit code 0
hanxiao commented 4 years ago

what if you use dry_run to test it? will it stop normally?

f = Flow().add(...)

with f:
   f.dry_run()
nan-wang commented 4 years ago

what if you use dry_run to test it? will it stop normally?

f = Flow().add(...)

with f:
   f.dry_run()

No, it got stuck at the last step.

/Users/nanwang/.pyenv/versions/3.7.5/bin/python3.7 /Users/nanwang/Codes/jina-ai/examples/flower-search/app.py -t index -n 20
           JINA@2632[W]:parser FlowPod can not recognize the following args: ['--seperated-workspace'], they are ignored. if you are using them from a global args (e.g. Flow), then please ignore this message
           Flow@2632[S]:successfully built Flow from a yaml config
    ImageReader@2635[I]:post initiating, this may take some time...
    ImageReader@2635[I]:post initiating, this may take some time takes 0.001 secs
    ImageReader@2635[S]:successfully built ImageReader from a yaml config
         loader@2635[I]:setting up sockets...
         loader@2635[I]:input tcp://0.0.0.0:54797 (PULL_BIND)    output tcp://0.0.0.0:54798 (PUB_BIND)   control over tcp://0.0.0.0:54799 (PAIR_BIND)
         loader@2635[S]:ready and listening
   ImageFlipper@2636[I]:post initiating, this may take some time...
   ImageFlipper@2636[I]:post initiating, this may take some time takes 0.000 secs
   ImageFlipper@2636[S]:successfully built ImageFlipper from a yaml config
        flipper@2636[I]:setting up sockets...
        flipper@2636[I]:input tcp://0.0.0.0:54798 (SUB_CONNECT)      output tcp://0.0.0.0:54805 (PUSH_CONNECT)   control over tcp://0.0.0.0:54803 (PAIR_BIND)
        flipper@2636[S]:ready and listening
ImageNormalizer@2637[I]:post initiating, this may take some time...
ImageNormalizer@2637[I]:post initiating, this may take some time takes 0.000 secs
ImageNormalizer@2637[S]:successfully built ImageNormalizer from a yaml config
     normalizer@2637[I]:setting up sockets...
     normalizer@2637[I]:input tcp://0.0.0.0:54805 (PULL_BIND)    output tcp://0.0.0.0:54809 (PUSH_CONNECT)   control over tcp://0.0.0.0:54807 (PAIR_BIND)
     normalizer@2637[S]:ready and listening
   BaseExecutor@2638[I]:post initiating, this may take some time...
   BaseExecutor@2638[I]:post initiating, this may take some time takes 0.000 secs
   BaseExecutor@2638[S]:successfully built BaseExecutor from a yaml config
   encoder-head@2638[I]:setting up sockets...
   encoder-head@2638[I]:input tcp://0.0.0.0:54809 (PULL_BIND)    output tcp://0.0.0.0:54814 (ROUTER_BIND)    control over tcp://0.0.0.0:54813 (PAIR_BIND)
   encoder-head@2638[S]:ready and listening
   BaseExecutor@2639[I]:post initiating, this may take some time...
   BaseExecutor@2639[I]:post initiating, this may take some time takes 0.000 secs
   BaseExecutor@2639[S]:successfully built BaseExecutor from a yaml config
   encoder-tail@2639[I]:setting up sockets...
   encoder-tail@2639[I]:input tcp://0.0.0.0:54815 (PULL_BIND)    output tcp://host.docker.internal:54819 (PUSH_CONNECT)  control over tcp://0.0.0.0:54816 (PAIR_BIND)
   encoder-tail@2639[S]:ready and listening
      encoder-1@2640[I]:waiting ready signal from the container
🐳 JINA@ 1[I]:
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMNNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMWxxxxxxxxxOMMMMMNxxxxxxxxx0MMMMMKddddddxkKWMMMMMMMMMMMMXOxdddxONMMMM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllo0MMMMMMMM0olllllllllo0MM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOlllllllllllloWMMMMMdllllllllllllldM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllllloMMMM0lllllllllllllllK
🐳 MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllllKMMM0lllllllllllllllO
🐳 MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMollllllllllllllO
🐳 MWOkkkkk0MMMKlllllllllkMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMxlllllllllllllO
🐳 NkkkkkkkkkMMKlllllllloMMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMMWOdolllllllllO
🐳 KkkkkkkkkkNMKllllllldMMMMMMMMWWWWWWWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MOkkkkkkk0MMKllllldXMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMWX00KXMMMMXxk0XMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 
🐳 ▶️  /opt/conda/bin/jina pod --yaml-path ImageTorchEncoder --name encoder --identity af77744cca --port-in 54814 --port-out 54815 --host-in host.docker.internal --host-out host.docker.internal --socket-in DEALER_CONNECT --socket-out PUSH_CONNECT --port-ctrl 54817 --timeout-ready 60000 --read-only --replica-id 1 --port-grpc 54812
🐳 array-in-pb = False
🐳 check-version = False
🐳 cli = pod
🐳 ctrl-with-ipc = False
🐳 dump-interval = 240
🐳 entrypoint = None
🐳 exit-no-dump = False
🐳 host = 0.0.0.0
🐳 🔧️                        host-in = host.docker.internal          
🐳 🔧️                       host-out = host.docker.internal          
🐳 🔧️                       identity = af77744cca                    
🐳 image = None
🐳 log-profile = False
🐳 log-remote = False
🐳 log-sse = False
🐳 logserver = False
🐳 logserver-config = /opt/conda/lib/python3.7/site-
🐳 max-idle-time = 60
🐳 max-message-size = -1
🐳 memory-hwm = -1
🐳 🔧️                           name = encoder                       
🐳 num-part = 1
🐳 override-exec-log = False
🐳 polling = ANY
🐳 🔧️                      port-ctrl = 54817                         
🐳 🔧️                      port-grpc = 54812                         
🐳 🔧️                        port-in = 54814                         
🐳 🔧️                       port-out = 54815                         
🐳 proxy = False
🐳 pull-latest = False
🐳 py-modules = None
🐳 🔧️                      read-only = True                          
🐳 reducing-yaml-path = _forward
🐳 🔧️                     replica-id = 1                             
🐳 replicas = 1
🐳 runtime = process
🐳 scheduling = LOAD_BALANCE
🐳 separated-workspace = False
🐳 shutdown-idle = False
🐳 🔧️                      socket-in = DEALER_CONNECT                
🐳 🔧️                     socket-out = PUSH_CONNECT                  
🐳 timeout = -1
🐳 timeout-ctrl = 5000
🐳 🔧️                  timeout-ready = 60000                         
🐳 volumes = None
🐳 🔧️                      yaml-path = ImageTorchEncoder             
🐳 
🐳 ImageTorchEncod@ 8[I]:post initiating, this may take some time...
🐳 ImageTorchEncod@ 8[I]:post initiating, this may take some time takes 0.667 secs
🐳 ImageTorchEncod@ 8[S]:successfully built ImageTorchEncoder from a yaml config
🐳 encoder@ 8[I]:setting up sockets...
🐳 encoder@ 8[I]:input tcp://host.docker.internal:54814 (DEALER_CONNECT)      output tcp://host.docker.internal:54815 (PUSH_CONNECT)  control over tcp://0.0.0.0:54817 (PAIR_BIND)
   encoder-head@2638[I]:received "control" from encoder▸⚐
      encoder-1@2640[S]:ready and listening
🐳 encoder@ 8[S]:ready and listening
      encoder-2@2641[I]:waiting ready signal from the container
🐳 JINA@ 1[I]:
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMNNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMMMMMMMMMMMWxxxxxxxxxOMMMMMNxxxxxxxxx0MMMMMKddddddxkKWMMMMMMMMMMMMXOxdddxONMMMM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllo0MMMMMMMM0olllllllllo0MM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOlllllllllllloWMMMMMdllllllllllllldM
🐳 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllllloMMMM0lllllllllllllllK
🐳 MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllllKMMM0lllllllllllllllO
🐳 MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMollllllllllllllO
🐳 MWOkkkkk0MMMKlllllllllkMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMxlllllllllllllO
🐳 NkkkkkkkkkMMKlllllllloMMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMMWOdolllllllllO
🐳 KkkkkkkkkkNMKllllllldMMMMMMMMWWWWWWWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MOkkkkkkk0MMKllllldXMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 MMWX00KXMMMMXxk0XMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
🐳 
🐳 ▶️  /opt/conda/bin/jina pod --yaml-path ImageTorchEncoder --name encoder --identity 65df7142dc --port-in 54814 --port-out 54815 --host-in host.docker.internal --host-out host.docker.internal --socket-in DEALER_CONNECT --socket-out PUSH_CONNECT --port-ctrl 54818 --timeout-ready 60000 --read-only --replica-id 2 --port-grpc 54812
🐳 array-in-pb = False
🐳 check-version = False
🐳 cli = pod
🐳 ctrl-with-ipc = False
🐳 dump-interval = 240
🐳 entrypoint = None
🐳 exit-no-dump = False
🐳 host = 0.0.0.0
🐳 🔧️                        host-in = host.docker.internal          
🐳 🔧️                       host-out = host.docker.internal          
🐳 🔧️                       identity = 65df7142dc                    
🐳 image = None
🐳 log-profile = False
🐳 log-remote = False
🐳 log-sse = False
🐳 logserver = False
🐳 logserver-config = /opt/conda/lib/python3.7/site-
🐳 max-idle-time = 60
🐳 max-message-size = -1
🐳 memory-hwm = -1
🐳 🔧️                           name = encoder                       
🐳 num-part = 1
🐳 override-exec-log = False
🐳 polling = ANY
🐳 🔧️                      port-ctrl = 54818                         
🐳 🔧️                      port-grpc = 54812                         
🐳 🔧️                        port-in = 54814                         
🐳 🔧️                       port-out = 54815                         
🐳 proxy = False
🐳 pull-latest = False
🐳 py-modules = None
🐳 🔧️                      read-only = True                          
🐳 reducing-yaml-path = _forward
🐳 🔧️                     replica-id = 2                             
🐳 replicas = 1
🐳 runtime = process
🐳 scheduling = LOAD_BALANCE
🐳 separated-workspace = False
🐳 shutdown-idle = False
🐳 🔧️                      socket-in = DEALER_CONNECT                
🐳 🔧️                     socket-out = PUSH_CONNECT                  
🐳 timeout = -1
🐳 timeout-ctrl = 5000
🐳 🔧️                  timeout-ready = 60000                         
🐳 volumes = None
🐳 🔧️                      yaml-path = ImageTorchEncoder             
🐳 
🐳 ImageTorchEncod@ 9[I]:post initiating, this may take some time...
🐳 ImageTorchEncod@ 9[I]:post initiating, this may take some time takes 0.690 secs
🐳 ImageTorchEncod@ 9[S]:successfully built ImageTorchEncoder from a yaml config
🐳 encoder@ 9[I]:setting up sockets...
🐳 encoder@ 9[I]:input tcp://host.docker.internal:54814 (DEALER_CONNECT)      output tcp://host.docker.internal:54815 (PUSH_CONNECT)  control over tcp://0.0.0.0:54818 (PAIR_BIND)
   encoder-head@2638[I]:received "control" from encoder▸⚐
      encoder-2@2641[S]:ready and listening
🐳 encoder@ 9[S]:ready and listening
   NumpyIndexer@2642[I]:post initiating, this may take some time...
   NumpyIndexer@2642[I]:post initiating, this may take some time takes 0.001 secs
   NumpyIndexer@2642[S]:successfully built NumpyIndexer from a yaml config
  BasePbIndexer@2642[I]:post initiating, this may take some time...
  BasePbIndexer@2642[I]:post initiating, this may take some time takes 0.000 secs
  BasePbIndexer@2642[S]:successfully built BasePbIndexer from a yaml config
   ChunkIndexer@2642[I]:post initiating, this may take some time...
   ChunkIndexer@2642[I]:post initiating, this may take some time takes 0.000 secs
   ChunkIndexer@2642[S]:successfully built ChunkIndexer from a yaml config
   NumpyIndexer@2642[W]:you can not query from <jina.executors.indexers.vector.numpy.NumpyIndexer object at 0x117441650> as its "query_handler" is not set. If you are indexing data then that is fine, just means you can not do querying-while-indexing.If you are querying data then the index file must be broken.
  chunk_indexer@2642[I]:setting up sockets...
  chunk_indexer@2642[I]:input tcp://0.0.0.0:54819 (PULL_BIND)    output tcp://0.0.0.0:54827 (PUSH_CONNECT)   control over tcp://0.0.0.0:54821 (PAIR_BIND)
  chunk_indexer@2642[S]:ready and listening
 LeveldbIndexer@2643[I]:post initiating, this may take some time...
 LeveldbIndexer@2643[I]:post initiating, this may take some time takes 0.000 secs
 LeveldbIndexer@2643[S]:restore LeveldbIndexer from /tmp/jina/flower/dtwyuqhi/LeveldbIndexer.bin
    doc_indexer@2643[I]:setting up sockets...
    doc_indexer@2643[I]:input tcp://0.0.0.0:54798 (SUB_CONNECT)      output tcp://0.0.0.0:54827 (PUSH_CONNECT)   control over tcp://0.0.0.0:54825 (PAIR_BIND)
    doc_indexer@2643[S]:ready and listening
   BaseExecutor@2644[I]:post initiating, this may take some time...
   BaseExecutor@2644[I]:post initiating, this may take some time takes 0.000 secs
   BaseExecutor@2644[S]:successfully built BaseExecutor from a yaml config
       join_all@2644[I]:setting up sockets...
       join_all@2644[I]:input tcp://0.0.0.0:54827 (PULL_BIND)    output tcp://0.0.0.0:54828 (PUSH_BIND)  control over tcp://0.0.0.0:54829 (PAIR_BIND)
       join_all@2644[S]:ready and listening
   BaseExecutor@2632[I]:post initiating, this may take some time...
   BaseExecutor@2632[I]:post initiating, this may take some time takes 0.001 secs
     GatewayPea@2632[S]:gateway is listening at: 0.0.0.0:54834
           Flow@2632[I]:8 Pods (i.e. 11 Peas) are running in this Flow
           Flow@2632[S]:flow is now ready for use, current build_level is GRAPH
       PyClient@2632[S]:connected to the gateway at 0.0.0.0:54834!
        gateway@2632[I]:setting up sockets...
        gateway@2632[I]:input tcp://0.0.0.0:54828 (PULL_CONNECT)     output tcp://0.0.0.0:54797 (PUSH_CONNECT)   control over ipc:///var/folders/6z/fwt6w1tx56nfk69ry1p265zc0000gp/T/tmp3bq4eyez (PAIR_BIND)
        gateway@2632[I]:prefetching 50 requests...
        gateway@2632[W]:if this takes too long, you may want to take smaller "--prefetch" or ask client to reduce "--batch-size"
        gateway@2632[I]:prefetching 50 requests takes 0.001 secs
        gateway@2632[I]:send: 0 recv: 0 pending: 0
         loader@2635[I]:received "control" from gateway▸⚐
        flipper@2636[I]:received "control" from gateway▸loader▸⚐
    doc_indexer@2643[I]:received "control" from gateway▸loader▸⚐
     normalizer@2637[I]:received "control" from gateway▸loader▸flipper▸⚐
   encoder-head@2638[I]:received "control" from gateway▸loader▸flipper▸normalizer▸⚐
🐳 encoder@ 8[I]:received "control" from gateway▸loader▸flipper▸normalizer▸encoder-head▸⚐
   encoder-tail@2639[I]:received "control" from gateway▸loader▸flipper▸normalizer▸encoder-head▸encoder▸⚐
   encoder-head@2638[I]:received "control" from gateway▸loader▸flipper▸normalizer▸encoder-head▸encoder▸⚐
       join_all@2644[I]:received "control" from gateway▸loader▸doc_indexer▸⚐
nan-wang commented 4 years ago

Now this bug can be reproduced with the flower-search demo at https://github.com/jina-ai/examples/tree/master/flower-search. Replace the #Replicas at flow-index.yml#L19 with any number larger than 1 will reproduce this bug.

nan-wang commented 4 years ago

check out https://github.com/jina-ai/examples/tree/master/x-as-service

nan-wang commented 4 years ago

Based on the following cases, I guess the problem is due to the wrong setup in the tail Pea when the Pod uses ContainerPea. In this case, the tail get inputs from tcp://0.0.0.0 and outputs to host.docker.internal, which is an internal Docker host. This leads to the problem of unresolvable host. 

   encoder-tail@3508[I]:input tcp://0.0.0.0:54326 (PULL_BIND)    output tcp://0.0.0.0:54321 (PUSH_BIND)  control over tcp://0.0.0.0:54327 (PAIR_BIND)
encoder-tail@2341[I]:input tcp://0.0.0.0:61417 (PULL_BIND)   output tcp://host.docker.internal:61421 (PUSH_CONNECT)  control over tcp://0.0.0.0:61418 (PAIR_BIND)
chunk_indexer@2345[I]:input tcp://0.0.0.0:61421 (PULL_BIND)      output tcp://0.0.0.0:61422 (PUSH_BIND)  control over tcp://0.0.0.0:61423 (PAIR_BIND)
encoder-tail@2566[I]:input tcp://0.0.0.0:49171 (PULL_BIND)   output tcp://0.0.0.0:49176 (PUSH_CONNECT)   control over tcp://0.0.0.0:49172 (PAIR_BIND)
chunk_indexer@2569[I]:input tcp://0.0.0.0:49176 (PULL_BIND)      output tcp://0.0.0.0:49177 (PUSH_BIND)  control over tcp://0.0.0.0:49178 (PAIR_BIND)
```bash

- 使用单encoder，encoder使用host.docker.internal输出，chunk_indexer使用0.0.0.0侦听，正常运行

```bash
🐳     encoder@ 8[I]:input tcp://0.0.0.0:62095 (PULL_BIND)    output tcp://host.docker.internal:62099 (PUSH_CONNECT)  control over tcp://0.0.0.0:62097 (PAIR_BIND)
chunk_indexer@2407[I]:input tcp://0.0.0.0:62099 (PULL_BIND)      output tcp://0.0.0.0:62100 (PUSH_BIND)  control over tcp://0.0.0.0:62101 (PAIR_BIND)
    encode-tail@3420[I]:input tcp://0.0.0.0:52296 (PULL_BIND)    output tcp://0.0.0.0:52291
nan-wang commented 4 years ago

When bind_docker == True, the returning depends on whether the connect is a tail or not. If it is a tail, the returning should be __default_host__ or else it should be local_host.

def _fill_in_host(bind_args, connect_args):
    from sys import platform

    bind_local = (bind_args.host == '0.0.0.0')
    bind_docker = (bind_args.image is not None and bind_args.image)
    conn_local = (connect_args.host == '0.0.0.0')
    conn_docker = (connect_args.image is not None and connect_args.image)
    bind_conn_same_remote = not bind_local and not conn_local and (bind_args.host == connect_args.host)
    if platform == "linux" or platform == "linux2":
        local_host = '0.0.0.0'
    else:
        local_host = 'host.docker.internal'

    if bind_local and conn_local and conn_docker:
        if bind_docker:
            return local_host
        else:
            return __default_host__
    elif bind_local and conn_local and not conn_docker:
        return __default_host__
    elif not bind_local and bind_conn_same_remote:
        if conn_docker:
            return local_host
        else:
            return __default_host__
    else:
        return bind_args.host