jina-ai / executors

internal-only
Apache License 2.0
31 stars 12 forks source link

Bug FaissPostgress - int not parsed #208

Closed florian-hoenicke closed 2 years ago

florian-hoenicke commented 2 years ago

When defining the pod via yaml, the shards and replicas are interpreted as string. This leads to the exception bellow:

Config

- name: text_indexer
    shards: 1
    uses: 'jinahub+docker://FaissPostgresSearcher'
    uses_with:
      startup_sync_args:
        only_delta: true
      total_shards: shards
      hostname: postgres.postgres.svc.cluster.local
      username: postgresadmin
      database: postgresdb
      table: text_indexer

Error

File "/usr/local/lib/python3.7/site-packages/jinahub/indexers/searcher/FaissSearcher/faiss_searcher.py", line 129, in __init__
    self._load_dump(dump_path, dump_func, prefetch_size, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/jinahub/indexers/searcher/FaissSearcher/faiss_searcher.py", line 145, in _load_dump
    iterator = dump_func(shard_id=self.runtime_args.pea_id)
  File "/usr/local/lib/python3.7/site-packages/jinahub/indexers/storage/PostgreSQLStorage/postgres_indexer.py", line 264, in _get_delta
    shard_id, total_shards, self.virtual_shards
  File "/usr/local/lib/python3.7/site-packages/jinahub/indexers/storage/PostgreSQLStorage/postgres_indexer.py", line 239, in _vshards_to_get
    if shard_id > total_shards - 1:
TypeError: unsupported operand type(s) for -: 'str' and 'int'
cristianmtr commented 2 years ago

Can u try setting the

      total_shards: shards

to a number, instead of shards?