Multiple modules/files for custom executor

[janandreschweiger]

Hey everyone,

we have build a very complicated executor and want to split the source code into different files/modules. I would expect that it works like for the MongoDBIndexer.

I tried various ways, but I always get one of the following two errors:

• ImportError('attempted relative import with no known parent package')
• ModuleNotFoundError("No module named 'segment'")

It would be cool if someone could have a look on this. Thank you!

[JoanFM]

Hey everyone,

we have build a very complicated executor and want to split the source code into different files/modules. I would expect that it works like for the MongoDBIndexer.

I tried various ways, but I always get one of the following two errors:

• ImportError('attempted relative import with no known parent package')
• ModuleNotFoundError("No module named 'segment'")

It would be cool if someone could have a look on this. Thank you!

Hey @janandreschweiger ,

thanks for opening the issue, could you share some sample code or structure that can help us take a look at it?

[janandreschweiger]

Sure! Thanks @JoanFM for your help.

crafter.yml

!CustomCrafter
metas:
  py_modules: custom_crafter.py # I also tried to add helper.py here

custom_crafter.py

class CustomCrafter(BaseSegmenter):
    def craft(self, buffer: bytes, *args, **kwargs):
        from helper import helper_function
        # or:
        from .helper import helper_function
        # additional code

helper.py

def helper_function():
    pass

Structure:

flows/
  search-flow.yml
pods/
  crafter.yml
  helper.py
  custom_crafter.py

[bhavsarpratik]

Hey @janandreschweiger, thanks for trying Jina! You can have a look at this example on how we use a custom executor with extra helper files. You might be missing sys.path.append(".")

[hanxiao]

@janandreschweiger

not sure if you know it but py_modules can be a list, say if A depends on B, which depends on C then

py_modules:
  - C.py
  - B.py
  - A.py

https://docs.jina.ai/api/jina.executors.metas.html?highlight=py_modules#confval-py_modules

[janandreschweiger]

Thanks for your kind help @hanxiao @bhavsarpratik! I did it exactly like in your example. I tried it again today and it still doesn't work.

Do you have a unit test that ensures that this error isn't caused by the latest version? If not, I could provide you with a minimal working example.

[ace-kay-law-neo]

I get the same error. Do you think that it is caused by jina @bhavsarpratik?

[hanxiao]

also related to #1480

[hanxiao]

@janandreschweiger @ace-kay-law-neo Problem solved and rolled out in 0.9.2, py_modules usage can be found in https://github.com/jina-ai/jina-hub/blob/master/README.md#use-py_modules-to-import-multiple-files

The test corresponds to @janandreschweiger your case can be found in https://github.com/jina-ai/jina/blob/master/tests/integration/issues/github_1546/test_pymodules_import.py#L73

[janandreschweiger]

Thank you @hanxiao. You are the best. from jinahub.helper import helper_function did the trick.