add Python 3.9 support - Githubissues

Describe the feature

Jina can not run on Python 3.9. We need community support on this to point out the issues, the out-of-date dependencies on Python 3.9.

Your proposal

@alexcg1 as a kick-off, can you add your log here?

Environment

Screenshots

On running jina hello-world on jina 0.7.12 on mac. jina -vf also fails

/opt/miniconda3/envs/test/lib/python3.9/site-packages/jina/importer.py:130: UserWarning: theses modules or classes can not be imported ['jina.drivers.evaluate', 'jina.drivers.encode', 'jina.drivers.rank', 'jina.drivers.predict', 'jina.drivers.search', 'jina.drivers.index', 'jina.drivers.cache', 'jina.drivers.multimodal', 'jina.drivers.craft']. You can use `jina check` to list all executors and drivers
  warnings.warn(
/opt/miniconda3/envs/test/lib/python3.9/site-packages/jina/importer.py:130: UserWarning: theses modules or classes can not be imported ['jina.executors.crafters.BaseCrafter', 'jina.executors.crafters.BaseSegmenter', 'jina.executors.indexers.cache', 'jina.executors.indexers.keyvalue.BinaryPbIndexer', 'jina.executors.indexers.keyvalue.DataURIPbIndexer', 'jina.executors.indexers.keyvalue.UniquePbIndexer', 'jina.executors.encoders.frameworks.BaseEncoder', 'jina.executors.encoders.frameworks.BaseMindsporeEncoder', 'jina.executors.encoders.frameworks.BaseOnnxEncoder', 'jina.executors.encoders.frameworks.BasePaddleEncoder', 'jina.executors.encoders.frameworks.BaseTFEncoder', 'jina.executors.encoders.frameworks.BaseTorchEncoder', 'jina.executors.encoders.tfserving.BaseEncoder', 'jina.executors.encoders.tfserving.BaseTFServingClientEncoder', 'jina.executors.encoders.tfserving.UnaryTFServingClientEncoder', 'jina.executors.evaluators.rank.recall.BaseRankingEvaluator', 'jina.executors.evaluators.rank.recall.RecallEvaluator', 'jina.executors.encoders.BaseAudioEncoder', 'jina.executors.encoders.BaseEncoder', 'jina.executors.encoders.BaseImageEncoder', 'jina.executors.encoders.BaseNumericEncoder', 'jina.executors.encoders.BaseTextEncoder', 'jina.executors.encoders.BaseVideoEncoder', 'jina.executors.encoders.numeric.BaseNumericEncoder', 'jina.executors.encoders.numeric.TransformEncoder', 'jina.executors.evaluators.rank.BaseRankingEvaluator', 'jina.executors.indexers.BaseVectorIndexer', 'jina.executors.indexers.CompoundIndexer', 'jina.executors.indexers.UniqueVectorIndexer', 'jina.executors.indexers.vector.BaseNumpyIndexer', 'jina.executors.indexers.vector.BaseVectorIndexer', 'jina.executors.indexers.vector.NumpyIndexer', 'jina.executors.evaluators.rank.precision.BaseRankingEvaluator', 'jina.executors.evaluators.rank.precision.PrecisionEvaluator']. You can use `jina check` to list all executors and drivers
  warnings.warn(
/opt/miniconda3/envs/test/lib/python3.9/site-packages/jina/importer.py:134: UserWarning: due to the missing dependencies or bad implementations, ['jina.hub.crafters.audio.SlidingWindowAudioSlicer.BaseSegmenter', 'jina.hub.crafters.audio.SlidingWindowAudioSlicer.SlidingWindowAudioSlicer', 'jina.hub.crafters.image.ImageResizer.BaseCrafter', 'jina.hub.crafters.image.ImageResizer.ImageResizer', 'jina.hub.encoders.video.VideoTorchEncoder.BaseTorchEncoder', 'jina.hub.encoders.video.VideoTorchEncoder.BaseVideoEncoder', 'jina.hub.encoders.video.VideoTorchEncoder.VideoTorchEncoder', 'jina.hub.indexers.keyvalue.RedisDBIndexer.BinaryPbIndexer', 'jina.hub.indexers.keyvalue.RedisDBIndexer.RedisDBIndexer', 'jina.hub.encoders.nlp.TransformerTorchEncoder.BaseEncoder', 'jina.hub.encoders.nlp.TransformerTorchEncoder.TransformerTorchEncoder', 'jina.hub.indexers.vector.AnnoyIndexer.AnnoyIndexer', 'jina.hub.indexers.vector.AnnoyIndexer.BaseNumpyIndexer', 'jina.hub.encoders.video.VideoPaddleEncoder.BasePaddleEncoder', 'jina.hub.encoders.video.VideoPaddleEncoder.VideoPaddleEncoder', 'jina.hub.encoders.numeric.FeatureAgglomerationEncoder.FeatureAgglomerationEncoder', 'jina.hub.encoders.numeric.FeatureAgglomerationEncoder.TransformEncoder', 'jina.hub.crafters.audio.AudioNormalizer.AudioNormalizer', 'jina.hub.crafters.audio.AudioNormalizer.BaseCrafter', 'jina.hub.indexers.vector.NGTIndexer.BaseNumpyIndexer', 'jina.hub.indexers.vector.NGTIndexer.NGTIndexer', 'jina.hub.encoders.image.ImageKerasEncoder.BaseTFEncoder', 'jina.hub.encoders.image.ImageKerasEncoder.ImageKerasEncoder', 'jina.hub.evaluators.rank.AveragePrecision.AveragePrecisionEvaluator', 'jina.hub.evaluators.rank.AveragePrecision.BaseRankingEvaluator', 'jina.hub.encoders.image.CustomKerasImageEncoder.BaseTFEncoder', 'jina.hub.encoders.image.CustomKerasImageEncoder.CustomKerasImageEncoder', 'jina.hub.encoders.audio.ChromaPitchEncoder.BaseAudioEncoder', 'jina.hub.encoders.audio.ChromaPitchEncoder.ChromaPitchEncoder', 'jina.hub.encoders.numeric.IncrementalPCAEncoder.IncrementalPCAEncoder', 'jina.hub.encoders.numeric.IncrementalPCAEncoder.TransformEncoder', 'jina.hub.indexers.vector.NmsLibIndexer.BaseNumpyIndexer', 'jina.hub.indexers.vector.NmsLibIndexer.NmsLibIndexer', 'jina.hub.crafters.nlp.TikaExtractor.BaseCrafter', 'jina.hub.crafters.nlp.TikaExtractor.TikaExtractor', 'jina.hub.crafters.image.CenterImageCropper.BaseCrafter', 'jina.hub.crafters.image.CenterImageCropper.CenterImageCropper', 'jina.hub.crafters.audio.AudioSlicer.AudioSlicer', 'jina.hub.crafters.audio.AudioSlicer.BaseSegmenter', 'jina.hub.indexers.vector.FaissIndexer.BaseNumpyIndexer', 'jina.hub.indexers.vector.FaissIndexer.FaissIndexer', 'jina.hub.encoders.image.ImageTorchEncoder.BaseTorchEncoder', 'jina.hub.encoders.image.ImageTorchEncoder.ImageTorchEncoder', 'jina.hub.crafters.audio.AudioReader.AudioReader', 'jina.hub.crafters.audio.AudioReader.BaseCrafter', 'jina.hub.crafters.nlp.JiebaSegmenter.BaseSegmenter', 'jina.hub.crafters.nlp.JiebaSegmenter.JiebaSegmenter', 'jina.hub.evaluators.rank.NdcgEvaluator.BaseRankingEvaluator', 'jina.hub.evaluators.rank.NdcgEvaluator.NDCGEvaluator', 'jina.hub.encoders.image.CustomImageTorchEncoder.BaseTorchEncoder', 'jina.hub.encoders.image.CustomImageTorchEncoder.CustomImageTorchEncoder', 'jina.hub.crafters.image.RandomImageCropper.BaseSegmenter', 'jina.hub.crafters.image.RandomImageCropper.RandomImageCropper', 'jina.hub.encoders.numeric.RandomGaussianEncoder.RandomGaussianEncoder', 'jina.hub.encoders.numeric.RandomGaussianEncoder.TransformEncoder', 'jina.hub.evaluators.rank.f1ScoreEvaluator.BaseRankingEvaluator', 'jina.hub.evaluators.rank.f1ScoreEvaluator.f1ScoreEvaluator', 'jina.hub.crafters.nlp.Sentencizer.BaseSegmenter', 'jina.hub.crafters.nlp.Sentencizer.Sentencizer', 'jina.hub.crafters.image.TorchObjectDetectionSegmenter.BaseSegmenter', 'jina.hub.crafters.image.TorchObjectDetectionSegmenter.TorchObjectDetectionSegmenter', 'jina.hub.encoders.image.BigTransferEncoder.BaseTFEncoder', 'jina.hub.encoders.image.BigTransferEncoder.BigTransferEncoder', 'jina.hub.encoders.numeric.CompressionVaeEncoder.BaseNumericEncoder', 'jina.hub.encoders.numeric.CompressionVaeEncoder.BaseTFEncoder', 'jina.hub.encoders.numeric.CompressionVaeEncoder.CompressionVaeEncoder', 'jina.hub.encoders.nlp.FlairTextEncoder.BaseTorchEncoder', 'jina.hub.encoders.nlp.FlairTextEncoder.FlairTextEncoder', 'jina.hub.crafters.nlp.DeepSegmenter.BaseSegmenter', 'jina.hub.crafters.nlp.DeepSegmenter.DeepSegmenter', 'jina.hub.crafters.numeric.ArrayStringReader.ArrayStringReader', 'jina.hub.crafters.numeric.ArrayStringReader.BaseCrafter', 'jina.hub.crafters.numeric.ArrayBytesReader.ArrayBytesReader', 'jina.hub.crafters.numeric.ArrayBytesReader.BaseCrafter', 'jina.hub.encoders.nlp.UniversalSentenceEncoder.BaseTFEncoder', 'jina.hub.encoders.nlp.UniversalSentenceEncoder.UniversalSentenceEncoder', 'jina.hub.indexers.vector.MilvusIndexer.BaseVectorIndexer', 'jina.hub.indexers.vector.MilvusIndexer.MilvusIndexer', 'jina.hub.crafters.nlp.SlidingWindowSegmenter.BaseSegmenter', 'jina.hub.crafters.nlp.SlidingWindowSegmenter.SlidingWindowSegmenter', 'jina.hub.evaluators.rank.MeanReciprocalRankEvaluator.BaseRankingEvaluator', 'jina.hub.evaluators.rank.MeanReciprocalRankEvaluator.MeanReciprocalRankEvaluator', 'jina.hub.encoders.nlp.FarmTextEncoder.BaseTorchEncoder', 'jina.hub.encoders.nlp.FarmTextEncoder.FarmTextEncoder', 'jina.hub.evaluators.rank.ReciprocalRankEvaluator.BaseRankingEvaluator', 'jina.hub.evaluators.rank.ReciprocalRankEvaluator.ReciprocalRankEvaluator', 'jina.hub.indexers.vector.ScannIndexer.BaseNumpyIndexer', 'jina.hub.indexers.vector.ScannIndexer.ScannIndexer', 'jina.hub.crafters.nlp.PDFExtractorSegmenter.BaseSegmenter', 'jina.hub.crafters.nlp.PDFExtractorSegmenter.PDFExtractorSegmenter', 'jina.hub.encoders.nlp.OneHotTextEncoder.BaseTextEncoder', 'jina.hub.encoders.nlp.OneHotTextEncoder.OneHotTextEncoder', 'jina.hub.encoders.nlp.TextPaddlehubEncoder.BasePaddleEncoder', 'jina.hub.encoders.nlp.TextPaddlehubEncoder.TextPaddlehubEncoder', 'jina.hub.encoders.image.ImagePaddlehubEncoder.BasePaddleEncoder', 'jina.hub.encoders.image.ImagePaddlehubEncoder.ImagePaddlehubEncoder', 'jina.hub.crafters.image.ImageFlipper.BaseCrafter', 'jina.hub.crafters.image.ImageFlipper.ImageFlipper', 'jina.hub.indexers.keyvalue.LevelDBIndexer.BinaryPbIndexer', 'jina.hub.indexers.keyvalue.LevelDBIndexer.LevelDBIndexer', 'jina.hub.encoders.nlp.TransformerTFEncoder.BaseEncoder', 'jina.hub.encoders.nlp.TransformerTFEncoder.TransformerTFEncoder', 'jina.hub.encoders.nlp.VSETextEncoder', 'jina.hub.encoders.numeric.RandomSparseEncoder.RandomSparseEncoder', 'jina.hub.encoders.numeric.RandomSparseEncoder.TransformEncoder', 'jina.hub.crafters.image.ImageNormalizer.BaseCrafter', 'jina.hub.crafters.image.ImageNormalizer.ImageNormalizer', 'jina.hub.encoders.numeric.TSNEEncoder.BaseNumericEncoder', 'jina.hub.encoders.numeric.TSNEEncoder.TSNEEncoder', 'jina.hub.indexers.vector.ZarrIndexer.NumpyIndexer', 'jina.hub.indexers.vector.ZarrIndexer.ZarrIndexer', 'jina.hub.encoders.numeric.FastICAEncoder.FastICAEncoder', 'jina.hub.encoders.numeric.FastICAEncoder.TransformEncoder', 'jina.hub.indexers.keyvalue.MongoDBIndexer.BinaryPbIndexer', 'jina.hub.indexers.keyvalue.MongoDBIndexer.MongoDBIndexer', 'jina.hub.encoders.numeric.BaseNumericEncoder', 'jina.hub.encoders.numeric.TransformEncoder', 'jina.hub.crafters.image.SlidingWindowImageCropper.BaseSegmenter', 'jina.hub.crafters.image.SlidingWindowImageCropper.SlidingWindowImageCropper', 'jina.hub.encoders.image.VSEImageEncoder', 'jina.hub.crafters.image.FiveImageCropper.BaseSegmenter', 'jina.hub.crafters.image.FiveImageCropper.FiveImageCropper', 'jina.hub.indexers.vector.SptagIndexer.BaseNumpyIndexer', 'jina.hub.indexers.vector.SptagIndexer.SptagIndexer', 'jina.hub.encoders.nlp.LaserEncoder.BaseTorchEncoder', 'jina.hub.encoders.nlp.LaserEncoder.LaserEncoder', 'jina.hub.encoders.image.ImageOnnxEncoder.BaseOnnxEncoder', 'jina.hub.encoders.image.ImageOnnxEncoder.ImageOnnxEncoder', 'jina.hub.crafters.image.ImageCropper.BaseCrafter', 'jina.hub.crafters.image.ImageCropper.ImageCropper', 'jina.hub.crafters.audio.AudioMonophoner.AudioMonophoner', 'jina.hub.crafters.audio.AudioMonophoner.BaseCrafter', 'jina.hub.crafters.image.ImageReader.BaseCrafter', 'jina.hub.crafters.image.ImageReader.ImageReader', 'jina.hub.encoders.audio.MFCCTimbreEncoder.BaseAudioEncoder', 'jina.hub.encoders.audio.MFCCTimbreEncoder.ChromaPitchEncoder', 'jina.hub.encoders.audio.MFCCTimbreEncoder.MFCCTimbreEncoder', 'jina.hub.encoders.audio.Wav2VecSpeechEncoder.BaseAudioEncoder', 'jina.hub.encoders.audio.Wav2VecSpeechEncoder.BaseTorchEncoder', 'jina.hub.encoders.audio.Wav2VecSpeechEncoder.Wav2VecSpeechEncoder'] can not be imported if you are using these executors/drivers, they wont work. You can use `jina check` to list all executors and drivers
  warnings.warn(
Traceback (most recent call last):
  File "/opt/miniconda3/envs/test/bin/jina", line 8, in <module>
    sys.exit(main())
  File "/opt/miniconda3/envs/test/lib/python3.9/site-packages/cli/__init__.py", line 88, in main
    args = _get_run_args()
  File "/opt/miniconda3/envs/test/lib/python3.9/site-packages/cli/__init__.py", line 8, in _get_run_args
    from jina.logging import default_logger
  File "/opt/miniconda3/envs/test/lib/python3.9/site-packages/jina/__init__.py", line 148, in <module>
    from jina.types.request import Request
  File "/opt/miniconda3/envs/test/lib/python3.9/site-packages/jina/types/request/__init__.py", line 3, in <module>
    from ..sets import QueryLangSet, DocumentSet
  File "/opt/miniconda3/envs/test/lib/python3.9/site-packages/jina/types/sets/__init__.py", line 4, in <module>
    from .document_set import DocumentSet
  File "/opt/miniconda3/envs/test/lib/python3.9/site-packages/jina/types/sets/document_set.py", line 4, in <module>
    from google.protobuf.pyext._message import RepeatedCompositeContainer
ModuleNotFoundError: No module named 'google.protobuf.pyext._message'

This issue is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 4 days

Hi,

I am working on a PR for this issue.

I have found out the issue why jina fails to work on Python 3.9 I have resolved this issue partially on my local machine in a Docker container using the python:3.9-buster image, where I successfully get the output for running the command: jina -vf However, running the command: jina hello-world results in an error related to ruamel package. I'll be attaching screenshots for execution of both the commands. I'd appreciate to learn about your thoughts on how I can proceed for solving the error related to ruamel package.

jina -vf runs successfully now.

jina hello-world fails.

I built protobuf from source for cpp and python to help in resolving the error: ModuleNotFoundError: No module named 'google.protobuf.pyext._message' This error happens due to lack of official Python 3.9 support for Protobuf python and cpp implementations that work with PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION. I built protobuf and protobuf/python for both python and cpp from source using a fix, which helped resolve this issue. This allows jina -vf to run successfully.

To investigate the error encountered while running jina hello-world, I went through the code flow for the entire error stack trace. Copy pasting it here for ease of access:

Traceback (most recent call last):
  File "/usr/local/bin/jina", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/cli/__init__.py", line 90, in main
    getattr(api, args.cli.replace('-', '_'))(args)
  File "/usr/local/lib/python3.9/site-packages/cli/api.py", line 76, in hello_world
    hello_world(args)
  File "/usr/local/lib/python3.9/site-packages/jina/helloworld/__init__.py", line 52, in hello_world
    f = Flow.load_config(args.uses_index)
  File "/usr/local/lib/python3.9/site-packages/jina/flow/__init__.py", line 138, in load_config
    return yaml.load(fp)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/main.py", line 343, in load
    return constructor.get_single_data()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 113, in get_single_data
    return self.construct_document(node)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 118, in construct_document
    data = self.construct_object(node)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 181, in construct_non_recursive_object
    data = constructor(self, node)
  File "/usr/local/lib/python3.9/site-packages/jina/flow/__init__.py", line 99, in from_yaml
    return cls._get_instance_from_yaml(constructor, node)[0]
  File "/usr/local/lib/python3.9/site-packages/jina/flow/__init__.py", line 150, in _get_instance_from_yaml
    return get_parser(version=data.get('version', None)).parse(data), data
  File "/usr/local/lib/python3.9/site-packages/jina/flow/yaml_parser/v1.py", line 61, in parse
    obj = Flow(*tmp_a, env=envs, **tmp_p)
  File "/usr/local/lib/python3.9/site-packages/jina/flow/__init__.py", line 68, in __init__
    self.logger = JinaLogger(self.__class__.__name__, **vars(self.args))
  File "/usr/local/lib/python3.9/site-packages/jina/logging/logger.py", line 129, in __init__
    self.add_handlers(log_config, **context_vars)
  File "/usr/local/lib/python3.9/site-packages/jina/logging/logger.py", line 161, in add_handlers
    config = yaml.load(fp)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/main.py", line 343, in load
    return constructor.get_single_data()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
    node = self.composer.get_single_node()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 78, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 101, in compose_document
    node = self.compose_node(None, None)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 120, in compose_node
    anchor = event.anchor
AttributeError: 'NoneType' object has no attribute 'anchor'

This error occurs because of getting the value of event variable as None and then trying to access anchor attribute from a NoneType variable. I will need some help from you guys to understand more about how events are processed through the ruamel package in jina to work ahead on resolving this.

Hey @gamesterrishi ,

Thank you very much for your collaboration.

We are not doing any special thing in this case, we are trying to load python objects from the file https://github.com/jina-ai/jina/blob/master/jina/resources/logging.default.yml.

I am not sure if there is some syntax here that ruamel may not accept for python3.9?

Thanks @JoanFM for your comment. I had a similar initial assumption as you about thinking that ruamel might not be ready for Python 3.9. I investigated it further and checked the codebase to for relevant package wheels to get more idea about supported Python versions:

https://sourceforge.net/projects/ruamel-yaml/ https://sourceforge.net/p/ruamel-yaml/code/ci/default/tree/setup.py

https://www.wheelodex.org/projects/ruamel-yaml/ mentions ruamel.yaml.clib as one of the requirements.

Project: | ruamel.yaml Version: | 0.16.12 Filename: | ruamel.yaml-0.16.12-py2.py3-none-any.whl Uploaded: | 2020-09-04 14:26:26 +0000

Requires-Dist: | ruamel.yaml.clib (>=0.1.2); platform_python_implementation == "CPython" and python_version < "3.9"

https://www.wheelodex.org/projects/ruamel-yaml-clib/ shows a Python 3.9 wheel does exist.

Project: ruamel.yaml.clib Version: 0.2.2 Filename: ruamel.yaml.clib-0.2.2-cp39-cp39-manylinux2014_aarch64.whl Uploaded: | 2020-11-23 22:02:01 +0000

So I tried installing this wheel and executed jina hello-world again. The error that I was encountering got resolved and it ran properly and printed all the logs too.

However, the only issue which I now face is that I get the following line in the log: flow is closed and all resources should be released already, current build level is EMPTY instead of getting the usual line: flow is closed and all resources should be released already, current build level is 0

Any help around why the current build could be 0 is highly appreciated. I'll attach log ouput after doing some more investigation.

Thanks @JoanFM for your comment. I had a similar initial assumption as you about thinking that ruamel might not be ready for Python 3.9. I investigated it further and checked the codebase to for relevant package wheels to get more idea about supported Python versions:

https://sourceforge.net/projects/ruamel-yaml/ https://sourceforge.net/p/ruamel-yaml/code/ci/default/tree/setup.py

https://www.wheelodex.org/projects/ruamel-yaml/ mentions ruamel.yaml.clib as one of the requirements.

Project: | ruamel.yaml Version: | 0.16.12 Filename: | ruamel.yaml-0.16.12-py2.py3-none-any.whl Uploaded: | 2020-09-04 14:26:26 +0000

Requires-Dist: | ruamel.yaml.clib (>=0.1.2); platform_python_implementation == "CPython" and python_version < "3.9"

https://www.wheelodex.org/projects/ruamel-yaml-clib/ shows a Python 3.9 wheel does exist.

Project: ruamel.yaml.clib Version: 0.2.2 Filename: ruamel.yaml.clib-0.2.2-cp39-cp39-manylinux2014_aarch64.whl Uploaded: | 2020-11-23 22:02:01 +0000

So I tried installing this wheel and executed jina hello-world again. The error that I was encountering got resolved and it ran properly and printed all the logs too.

However, the only issue which I now face is that I get the following line in the log: flow is closed and all resources should be released already, current build level is EMPTY instead of getting the usual line: flow is closed and all resources should be released already, current build level is 0

Any help around why the current build could be 0 is highly appreciated. I'll attach log ouput after doing some more investigation.

Hey @gamesterrishi

That is not a problem, EMPTY is the enum name whose value is 0, so no problem about it. I guess some change on how enum values are printed

Hey @gamesterrishi, thanks for solving this. You can look at FlowBuildLevel in enums.py. I think the solution is ready and you can create a PR.

Hey @JoanFM and @bhavsarpratik, thanks for your comments.

Before I submit my PR, I'd like to share my proposal for solving this issue and confirming if it's the best way for us to proceed towards solving it. In summary, jina is unable to support Python 3.9 currently due to lack of official support for Python 3.9 in two of its dependencies: protobuf and raumel

Installing the latest wheel for ruamel.yaml.clib solves the ruamel issue.

As per my screenshots above, making use of the following PR allows us to successfully make protobuf work for Python 3.9: https://github.com/protocolbuffers/protobuf/pull/8121 Cloning this PR and then building protobuf and protobuf/python for both cpp and python allows us to make protobuf work with Python 3.9. However, since it's not officially merged yet and there is no news currently about when protobuf would officially release a new stable version for supporting Python 3.9, I believe that using this fix for solving the protobuf compatibility issue should not constitute towards an official stable release version in jina right now, since official releases of jina should include all dependencies to be picked only from stable release versions and not unofficial fixes. We should rather include this as a fix in the form of a docker image with a suitable label such as 'jina:python-3.9' or something similar and mention it accordingly in the documentation. Also, if we want to allow non Docker jina installations to function with Python 3.9, we could share a link in the documentation containing all the steps required by the user to implement this protobuf fix on their machine, or even share a script to solve it. I believe then we could wait for the official stable version from protobuf incorporating this fix to be released and use it while generating official releases for jina .

Please let me know your thoughts on this, and then I can proceed accordingly. Thanks.

Hey @JoanFM and @bhavsarpratik, thanks for your comments.

Before I submit my PR, I'd like to share my proposal for solving this issue and confirming if it's the best way for us to proceed towards solving it. In summary, jina is unable to support Python 3.9 currently due to lack of official support for Python 3.9 in two of its dependencies: protobuf and raumel

Installing the latest wheel for ruamel.yaml.clib solves the ruamel issue.

As per my screenshots above, making use of the following PR allows us to successfully make protobuf work for Python 3.9: protocolbuffers/protobuf#8121 Cloning this PR and then building protobuf and protobuf/python for both cpp and python allows us to make protobuf work with Python 3.9. However, since it's not officially merged yet and there is no news currently about when protobuf would officially release a new stable version for supporting Python 3.9, I believe that using this fix for solving the protobuf compatibility issue should not constitute towards an official stable release version in jina right now, since official releases of jina should include all dependencies to be picked only from stable release versions and not unofficial fixes. We should rather include this as a fix in the form of a docker image with a suitable label such as 'jina:python-3.9' or something similar and mention it accordingly in the documentation. Also, if we want to allow non Docker jina installations to function with Python 3.9, we could share a link in the documentation containing all the steps required by the user to implement this protobuf fix on their machine, or even share a script to solve it. I believe then we could wait for the official stable version from protobuf incorporating this fix to be released and use it while generating official releases for jina .

Please let me know your thoughts on this, and then I can proceed accordingly. Thanks.

Hey @gamesterrishi,

Thank you very much for the investigation. This is very helpful to understand clearly where we stand with this issue.

I think the idea of supporting a different tag for this without official protobuf or ruamel support is a good idea, but it would put too much mantainance cost on our side. I would rather wait for protobuf to have a stable version supporting it.

As per ruamel, we already opened an issue #1471 to consider migrating to another yaml parser solution that supports 3.9. So putting more effort on adaptingruamel version does not seem sustainable in the long run.

I think a good step forward would be to work on the migration in #1471

Hey @JoanFM , thanks for your sharing your thoughts. I was in fact looking for the differences between ruamel and PyYAML while trying to resolve the ruamel issue, since I have been solely making use of PyYAML for handling any yaml related operations in Python in my work, and was curious to know as to why ruamel was being used in jina instead of PyYAML. The following command was sufficient for me to resolve the ruamel compatibility: pip install --upgrade ruamel.yaml.clib which internally downloads the following wheel as shared above: ruamel.yaml.clib-0.2.2-cp39-cp39-manylinux2014_aarch64.whl

This shows that the latest version for ruamel.yaml.clib is officially supported, however the latest ruamel package does not include this latest version as its subdependency. I am not sure if installing a subdependency for a jina dependency separately should be considered as unstable or not. I just wanted to point this out before moving onto another issue. I totally agree with how it would be for the best to proceed ahead with PyYAML as it looks like a more stable option currently for the long run.

Also, attaching some screenshots too.

was curious to know as to why ruamel was being used in jina instead of PyYAML

To be fair, ruamel.yaml supports YAML 1.2 spec whereas PyYAML only supports YAML 1.1.

the PyYAML part is done in #1495

but it seems that Protobuf does not support Python 3.9 yet. I tested locally in docker and it gives

Traceback (most recent call last):
  File "/usr/local/bin/jina", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/cli/__init__.py", line 88, in main
    args = _get_run_args()
  File "/usr/local/lib/python3.9/site-packages/cli/__init__.py", line 8, in _get_run_args
    from jina.logging import default_logger
  File "/usr/local/lib/python3.9/site-packages/jina/__init__.py", line 148, in <module>
    from jina.types.request import Request
  File "/usr/local/lib/python3.9/site-packages/jina/types/request/__init__.py", line 5, in <module>
    from ..sets import QueryLangSet, DocumentSet
  File "/usr/local/lib/python3.9/site-packages/jina/types/sets/__init__.py", line 4, in <module>
    from .document import DocumentSet
  File "/usr/local/lib/python3.9/site-packages/jina/types/sets/document.py", line 6, in <module>
    from google.protobuf.pyext._message import RepeatedCompositeContainer
ModuleNotFoundError: No module named 'google.protobuf.pyext._message'

Related issues:

@gamesterrishi how did you manage to run hello-world in Python 3.9?

Hey @hanxiao,

Thanks for being so quick on the PyYAML migration implementation - it's amazing that now that jina will support PyYAML completely, since its a better and more stable choice in the long run as per the reasons that you have mentioned in: https://github.com/jina-ai/jina/issues/1471

I was in fact looking for the differences between ruamel and PyYAML while trying to resolve the ruamel issue

I came to know about it's differences with ruamel while working on this issue, as I initially felt that if there was no way to get ruamel working with Python 3.9, I would have to try on getting things working by replacing it with PyYAML.

https://yaml.readthedocs.io/en/latest/pyyaml.html

As per my screenshots above, making use of the following PR allows us to successfully make protobuf work for Python 3.9: protocolbuffers/protobuf#8121 Cloning this PR and then building protobuf and protobuf/python for both cpp and python allows us to make protobuf work with Python 3.9. However, since it's not officially merged yet and there is no news currently about when protobuf would officially release a new stable version for supporting Python 3.9, I believe that using this fix for solving the protobuf compatibility issue should not constitute towards an official stable release version in jina right now, since official releases of jina should include all dependencies to be picked only from stable release versions and not unofficial fixes. We should rather include this as a fix in the form of a docker image with a suitable label such as 'jina:python-3.9' or something similar and mention it accordingly in the documentation. Also, if we want to allow non Docker jina installations to function with Python 3.9, we could share a link in the documentation containing all the steps required by the user to implement this protobuf fix on their machine, or even share a script to solve it. I believe then we could wait for the official stable version from protobuf incorporating this fix to be released and use it while generating official releases for jina .

I'll share the list of commands with you in accordance with this, which would allow you to test things in a Docker container. I'd be happy if we could then try to incorporate it as a temporary fix for early support of Python 3.9 in jina.

Hey @hanxiao,

Please find the list of commands: (Commands to be executed are bold formatted)

Run a Docker container for Python 3.9 in an interactive mode. (You can mount directory with -v if needed as per your convenience). docker run -it --name jina_py39 python:3.9-buster bash
Display information about the environment. uname -r cat /etc/os-release which python python -V pip list
Install jina using pip and run jina -vf, which gives an error. pip install jina pip list jina -vf ModuleNotFoundError: No module named 'google.protobuf.pyext._message'
Troubleshoot above error. python -c "from google.protobuf.pyext import _message" PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION="python" python -c "from google.protobuf.pyext import _message" PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION="cpp" python -c "from google.protobuf.pyext import _message"
protobuf does not have an official stable release compatible with Python 3.9 yet. So, we will try to incorporate a fix from another pull request. https://github.com/protocolbuffers/protobuf/pull/8121 (Relevant Issue) https://github.com/busunkim96/protobuf/tree/add-39 (Git Repo of fix which we are going to clone)

https://github.com/protocolbuffers/protobuf/blob/master/src/README.md (C++ Installation Instructions - Unix)

To build protobuf from source, the following tools are needed: autoconf automake libtool make g++ unzip On Ubuntu/Debian, you can install them with: (These components are already present in our container) apt-get install autoconf automake libtool curl make g++ unzip

Get the source by "git clone" of the git repository for the fix. Make sure you have also cloned the submodules and generated the configure script: git clone https://github.com/busunkim96/protobuf.git cd protobuf git submodule update --init --recursive ./autogen.sh

To build and install the C++ Protocol Buffer runtime and the Protocol Buffer compiler (protoc) execute the following: ./configure make make check make install ldconfig # refresh shared library cache.

cd ..

Check out contents of some folders. The protobuf that's already installed is present in /usr/local/lib/python3.9/site-packages/google/protobuf/ ls -ltrh protobuf/ ls -ltrh protobuf/python ls -ltrh protobuf/python/google/protobuf/ ls -ltrh /usr/local/lib/python3.9/site-packages/google/protobuf/

https://github.com/protocolbuffers/protobuf/tree/master/python (Python Installation Instructions - Unix) Install Python Protobuf cd protobuf/python protoc --version Build and run the tests: python setup.py build python setup.py test To build, test, and use the C++ implementation, you must first compile libprotobuf.so: (cd .. && make)

The pure python performance is slow. For better performance lets install python c++ implementation. To build the C++ implementation run: python setup.py build --cpp_implementation Then run the tests like so: python setup.py test --cpp_implementation

Now, we need to uninstall already existing protobuf installed by pip and copy the contents of our newly compiled protobuf to the target Python directory. cd / ls -ltrh /usr/local/lib/python3.9/site-packages/google pip uninstall protobuf ls -ltrh /usr/local/lib/python3.9/site-packages/google cp -r /protobuf/python/google/protobuf /usr/local/lib/python3.9/site-packages/google/ ls -ltrh /usr/local/lib/python3.9/site-packages/google ls -ltrh /usr/local/lib/python3.9/site-packages/google/protobuf

Confirm that jina -vf runs now. python -c "from google.protobuf.pyext import _message" PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION="python" python -c "from google.protobuf.pyext import _message" PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION="cpp" python -c "from google.protobuf.pyext import _message" jina -vf PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION="python" jina -vf PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION="cpp" jina -vf

Run jina hello-world, which causes an error. jina hello-world

usr/local/lib/python3.9/site-packages/jina/flow/yaml_parser/__init__.py:30: UserWarning: can not find parser for version: 1.0, fallback to parser for version: 1
warnings.warn(f'can not find parser for version: {version}, '
Traceback (most recent call last):
File "/usr/local/bin/jina", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/cli/__init__.py", line 90, in main
getattr(api, args.cli.replace('-', '_'))(args)
File "/usr/local/lib/python3.9/site-packages/cli/api.py", line 76, in hello_world
hello_world(args)
File "/usr/local/lib/python3.9/site-packages/jina/helloworld/__init__.py", line 52, in hello_world
f = Flow.load_config(args.uses_index)
File "/usr/local/lib/python3.9/site-packages/jina/flow/__init__.py", line 138, in load_config
return yaml.load(fp)
File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/main.py", line 343, in load
return constructor.get_single_data()
File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 113, in get_single_data
return self.construct_document(node)
File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 118, in construct_document
data = self.construct_object(node)
File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
data = self.construct_non_recursive_object(node)
File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 181, in construct_non_recursive_object
data = constructor(self, node)
File "/usr/local/lib/python3.9/site-packages/jina/flow/__init__.py", line 99, in from_yaml
return cls._get_instance_from_yaml(constructor, node)[0]
File "/usr/local/lib/python3.9/site-packages/jina/flow/__init__.py", line 150, in _get_instance_from_yaml
return get_parser(version=data.get('version', None)).parse(data), data
File "/usr/local/lib/python3.9/site-packages/jina/flow/yaml_parser/v1.py", line 61, in parse
obj = Flow(*tmp_a, env=envs, **tmp_p)
File "/usr/local/lib/python3.9/site-packages/jina/flow/__init__.py", line 68, in __init__
self.logger = JinaLogger(self.__class__.__name__, **vars(self.args))
File "/usr/local/lib/python3.9/site-packages/jina/logging/logger.py", line 129, in __init__
self.add_handlers(log_config, **context_vars)
File "/usr/local/lib/python3.9/site-packages/jina/logging/logger.py", line 161, in add_handlers
config = yaml.load(fp)
File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/main.py", line 343, in load
return constructor.get_single_data()
File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
node = self.composer.get_single_node()
File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 78, in get_single_node
document = self.compose_document()
File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 101, in compose_document
node = self.compose_node(None, None)
File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 120, in compose_node
anchor = event.anchor
AttributeError: 'NoneType' object has no attribute 'anchor'

Fix ruamel error by installing latest Python 3.9 wheel for ruamel.yaml.clib. jina hello-world runs successfully now. pip list pip install --upgrade ruamel.yaml.clib pip list jina hello-world

I understand that maintaining releases has to be entirely automated and manageable. But do let me know if we could work out something that's maintainable and works. I believe that including a special tag Docker image in jina Docker Hub registry can work out, or we can add these instructions in the docs for installation for users who want to make use of jina in Python 3.9. Any other suggestions are welcome.

This issue is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 4 days

jina-ai / serve

add Python 3.9 support #1253