jina-ai / jina

☁️ Build multimodal AI applications with cloud-native stack
https://docs.jina.ai
Apache License 2.0
20.99k stars 2.22k forks source link

bug: running flow on windows #5824

Closed florian-hoenicke closed 1 year ago

florian-hoenicke commented 1 year ago

On the latest master, jina is hanging when deploying the following flow:

jtype: Flow
with:
  port: 8080
  protocol: http
jcloud:
  version: 3.14.2.dev18
  labels:
    creator: microchain
  name: gptdeploy
executors:
  - name: printhelloexecutor4715887
    uses: jinaai+docker://auth0-unified-448f11965ce142b6/PrintHelloExecutor4715887:latest
    jcloud:
      resources:
        instance: C2
        capacity: spot

error:

C:\Users\hoenicke\jina\gptdeploy\venv\Scripts\python.exe C:\Users\hoenicke\jina\gptdeploy\gptdeploy.py run --path microservice 
Run a jina flow locally
⠋ Fetching auth0-unified-448f11965ce142b6/PrintHelloExecutor4715887 from Jina 
⠋ Fetching auth0-unified-448f11965ce142b6/PrintHelloExecutor4715887 from Jina 
Hub ...

WARNI… printhelloexecutor4715887/rep-0@18720                [04/24/23 10:36:57]
       <jina.orchestrate.pods.container.ContainerPod object                    
       at 0x000001E8A04D1350> timeout after waiting for                        
       600000ms, if your executor takes time to load, you                      
       may increase --timeout-ready                                            
🐳  Process terminated, the container fails to start, check the arguments or   
entrypoint                                                                     
ERROR  Flow@18720 Flow is aborted due to                    [04/24/23 10:36:59]
       ['printhelloexecutor4715887'] can not be started.                       
WARNI… gateway/rep-0@18720 Pod was forced to close after 1  [04/24/23 10:37:00]
       second. Graceful closing is not available on                            
       Windows.                                                                
Traceback (most recent call last):
  File "C:\Users\hoenicke\jina\gptdeploy\gptdeploy.py", line 6, in <module>
    main()
  File "C:\Users\hoenicke\jina\gptdeploy\venv\Lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hoenicke\jina\gptdeploy\venv\Lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\hoenicke\jina\gptdeploy\venv\Lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hoenicke\jina\gptdeploy\venv\Lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hoenicke\jina\gptdeploy\venv\Lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hoenicke\jina\gptdeploy\src\cli.py", line 39, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hoenicke\jina\gptdeploy\src\cli.py", line 84, in run
    Runner().run(path)
  File "C:\Users\hoenicke\jina\gptdeploy\src\options\run\runner.py", line 10, in run
    run_locally(executor_name, latest_version_path)
  File "C:\Users\hoenicke\jina\gptdeploy\src\apis\jina_cloud.py", line 204, in run_locally
    with flow:
  File "C:\Users\hoenicke\jina\gptdeploy\venv\Lib\site-packages\jina\orchestrate\orchestrator.py", line 14, in __enter__
    return self.start()
           ^^^^^^^^^^^^
  File "C:\Users\hoenicke\jina\gptdeploy\venv\Lib\site-packages\jina\orchestrate\flow\builder.py", line 33, in arg_wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hoenicke\jina\gptdeploy\venv\Lib\site-packages\jina\orchestrate\flow\base.py", line 1832, in start
    self._wait_until_all_ready()
  File "C:\Users\hoenicke\jina\gptdeploy\venv\Lib\site-packages\jina\orchestrate\flow\base.py", line 1975, in _wait_until_all_ready
    raise RuntimeFailToStart
jina.excepts.RuntimeFailToStart
JoanFM commented 1 year ago

can you please run with JINA_LOG_LEVEL=DEBUG?

florian-hoenicke commented 1 year ago

Here are the complete logs before the timeout is happening

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
Run a jina flow locally
⠋ Fetching auth0-unified-448f11965ce142b6/PrintTestExecutor2622 from Jina Hub 
⠙ Fetching auth0-unified-448f11965ce142b6/PrintTestExecutor2622 from Jina Hub 
⠹ Fetching auth0-unified-448f11965ce142b6/PrintTestExecutor2622 from Jina Hub 
⠹ Fetching auth0-unified-448f11965ce142b6/PrintTestExecutor2622 from Jina Hub 
...

🐳 will use Docker image: jinahub/1xum4h0g:644686d75dc4fb62032a1793            
DEBUG  gateway/rep-0@6372 adding connection for deployment  [04/24/23 22:57:35]
       printtestexecutor2622/heads/0 to                                        
       grpc://127.0.0.1:55538                                                  
DEBUG  gateway/rep-0@6372 connection for deployment                            
       printtestexecutor2622/heads/0 to                                        
       grpc://127.0.0.1:55538 added                                            
INFO:     Waiting for application startup.
DEBUG  gateway/rep-0@6372 Running GatewayRuntime warmup     [04/24/23 22:57:38]
DEBUG  gateway/rep-0@6372 starting warmup task for                             
       deployment printtestexecutor2622                                        
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)
DEBUG  gateway/rep-0@21116 ready and listening              [04/24/23 22:57:38]
🐳                                                                             
🐳                                                                             
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM…
🐳                                                                             
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM…
🐳                                                                             
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMNNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM…
🐳                                                                             
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM…
🐳                                                                             
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM…
🐳                                                                             
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM…
🐳                                                                             
MMMMMMMMMMMMWxxxxxxxxxOMMMMMNxxxxxxxxx0MMMMMKddddddxkKWMMMMMMMMMMMMXOxdddxONMM…
🐳                                                                             
MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllo0MMMMMMMM0olllllllllo0…
🐳                                                                             
MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOlllllllllllloWMMMMMdlllllllllllll…
🐳                                                                             
MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllllloMMMM0llllllllllllll…
🐳                                                                             
MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllllKMMM0llllllllllllll…
🐳                                                                             
MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMolllllllllllll…
🐳                                                                             
MWOkkkkk0MMMKlllllllllkMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMxllllllllllll…
🐳                                                                             
NkkkkkkkkkMMKlllllllloMMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMMWOdollllllll…
🐳                                                                             
KkkkkkkkkkNMKllllllldMMMMMMMMWWWWWWWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM…
🐳                                                                             
MOkkkkkkk0MMKllllldXMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM…
🐳                                                                             
MMWX00KXMMMMXxk0XMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM…
🐳                                                                             
🐳                                                                             
🐳 /usr/local/bin/jina executor --uses config.yml --name                       
🐳 printtestexecutor2622/rep-0 --workspace-id d6f76d5ee3f6442dae28eab7b091596e 
🐳 --extra-search-paths                                                        
🐳 C:\Users\hoenicke\jina\gptdeploy\microservice\PrintTestExecutor2622\0_\v1   
🐳 --polling ANY --native --host 127.0.0.1 --env {"OPENAI_API_KEY":            
🐳 "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"} --noblock-on-start   
--port                                                                         
🐳 55538 --port-monitoring 59671                                               
🐳                                                                             
╭───────────────────────┬─────────────────────────────────────────────────────…
🐳 │              Argument │ Value                                            …
│                                                                              
🐳                                                                             
├───────────────────────┼─────────────────────────────────────────────────────…
🐳 │      allow-concurrent │ False                                            …
│                                                                              
🐳 │                   cli │ executor                                         …
│                                                                              
🐳 │           compression │ None                                             …
│                                                                              
🐳 │       connection-list │ None                                             …
│                                                                              
🐳 │   disable-auto-volume │ False                                            …
│                                                                              
🐳 │         docker-kwargs │ None                                             …
│                                                                              
🐳 │            entrypoint │ None                                             …
│                                                                              
🐳 │                   env │ {'OPENAI_API_KEY':                               …
│                                                                              
🐳 │                       │                                                   
'sk-FjGRu4imV3nBukjgWE5pT3BlbkFJqzKgnUr2D07fRXJpAmr… │                         
🐳 │       env-from-secret │ None                                             …
│                                                                              
🐳 │    exit-on-exceptions │ []                                               …
│                                                                              
🐳 │    extra-search-paths │                                                   
['C:\\Users\\hoenicke\\jina\\gptdeploy\\microservic… │                         
🐳 │              floating │ False                                            …
│                                                                              
🐳 │    force-network-mode │ AUTO                                             …
│                                                                              
🐳 │          force-update │ False                                            …
│                                                                              
🐳 │                  gpus │ None                                             …
│                                                                              
🐳 │  grpc-channel-options │ None                                             …
│                                                                              
🐳 │   grpc-server-options │ None                                             …
│                                                                              
🐳 │                  host │ ['127.0.0.1']                                    …
│                                                                              
🐳 │  install-requirements │ False                                            …
│                                                                              
🐳 │         k8s-namespace │ None                                             …
│                                                                              
🐳 │            log-config │ default                                          …
│                                                                              
🐳 │               metrics │ False                                            …
│                                                                              
🐳 │ metrics-exporter-host │ None                                             …
│                                                                              
🐳 │ metrics-exporter-port │ None                                             …
│                                                                              
🐳 │            monitoring │ False                                            …
│                                                                              
🐳 │                  name │ printtestexecutor2622/rep-0                      …
│                                                                              
🐳 │                native │ True                                             …
│                                                                              
🐳 │             no-reduce │ False                                            …
│                                                                              
🐳 │      noblock-on-start │ True                                             …
│                                                                              
🐳 │     output-array-type │ None                                             …
│                                                                              
🐳 │              pod-role │ WORKER                                           …
│                                                                              
🐳 │               polling │ ANY                                              …
│                                                                              
🐳 │                  port │ [55538]                                          …
│                                                                              
🐳 │       port-monitoring │ [59671]                                          …
│                                                                              
🐳 │       prefer-platform │ None                                             …
│                                                                              
🐳 │              protocol │ [<ProtocolType.GRPC: 0>]                         …
│                                                                              
🐳 │            py-modules │ None                                             …
│                                                                              
🐳 │                 quiet │ False                                            …
│                                                                              
🐳 │           quiet-error │ False                                            …
│                                                                              
🐳 │                reload │ False                                            …
│                                                                              
🐳 │              replicas │ 1                                                …
│                                                                              
🐳 │               retries │ -1                                               …
│                                                                              
🐳 │           runtime-cls │ WorkerRuntime                                    …
│                                                                              
🐳 │              shard-id │ 0                                                …
│                                                                              
🐳 │                shards │ 1                                                …
│                                                                              
🐳 │          timeout-ctrl │ 60                                               …
│                                                                              
🐳 │         timeout-ready │ 600000                                           …
│                                                                              
🐳 │          timeout-send │ None                                             …
│                                                                              
🐳 │  traces-exporter-host │ None                                             …
│                                                                              
🐳 │  traces-exporter-port │ None                                             …
│                                                                              
🐳 │               tracing │ False                                            …
│                                                                              
🐳 │                  uses │ config.yml                                       …
│                                                                              
🐳 │    uses-after-address │ None                                             …
│                                                                              
🐳 │   uses-before-address │ None                                             …
│                                                                              
🐳 │ uses-dynamic-batching │ None                                             …
│                                                                              
🐳 │            uses-metas │ None                                             …
│                                                                              
🐳 │         uses-requests │ None                                             …
│                                                                              
🐳 │             uses-with │ None                                             …
│                                                                              
🐳 │               volumes │ None                                             …
│                                                                              
🐳 │             workspace │ None                                             …
│                                                                              
🐳 │          workspace-id │ d6f76d5ee3f6442dae28eab7b091596e                 …
│                                                                              
🐳                                                                             
╰───────────────────────┴─────────────────────────────────────────────────────…
🐳 INFO   printtestexecutor2622/rep-0@ 1 start server bound to  [04/24/23      
20:57:40]                                                                      
🐳        127.0.0.1:55538                                                      
🐳 INFO   printtestexecutor2622/rep-0@ 1 Executor               [04/24/23      
20:57:40]                                                                      
🐳        PrintTestExecutor2622 started                                        

Looks good but it is hanging until the timeout is raised

JoanFM commented 1 year ago

can you show me the code of this Executor?

JoanFM commented 1 year ago

while doing so, can u check if u can do jina ping executor grpc://0.0.0.0:port?