jina-ai / jina

☁️ Build multimodal AI applications with cloud-native stack
https://docs.jina.ai
Apache License 2.0
20.99k stars 2.22k forks source link

How to use the same executor multiple times in flow #4793

Closed tommykoctur closed 2 years ago

tommykoctur commented 2 years ago

Describe your proposal/problem I am using multiple local executors. One of them is LMDB. I would like to access LMDB twice in my flow, but without running two executors but just one. (to save RAM).

- name: chunk_match_db_filler
  uses:
    jtype: LMDBStorage
    py_modules:
      - executors/lmdb_storage.py
  uses-with:
    default_traversal_paths: ['cm']

- name: ranker
  uses:
    jtype: IRanker
    py_modules:
      - executors/ranker.py
  uses-with:
    dim: 768

- name: root_match_db_filler
  uses:
    jtype: LMDBStorage
    py_modules:
      - executors/lmdb_storage.py
  uses-with:
    default_traversal_paths: ['rm']

It will spin two lmdb_storage.py executors on different ports.

Environment

JoanFM commented 2 years ago

Hey @tommykoctur ,

You may want to follow this documentation:

https://docs.jina.ai/how-to/external-executor/?highlight=external+executor&utm_source=jina#how-to-use-external-executors