add python pkg : torchtext

shaheen-coder commented 1 month ago
Why is it worth to add this package?

As you know, torchtext is a popular library for natural language processing (NLP) tasks within the PyTorch ecosystem. Having torchtext readily available through pkg would significantly benefit Termux users working on NLP projects. While I understand there might not be a pre-built package yet, I'd be happy to contribute any information or assistance that might help with the packaging process
Home page URL

No response
Source code URL

https://github.com/pytorch/text
Packaging policy acknowledgement

[x] The project is actively developed.
[X] The project has existing packages and is "well known".
[X] Licensed under an open source license.
[ ] Not available through a language package manager: pip, npm, cpan, cargo, etc.
[x] Not taking up too much disk space (< 100MiB per architecture, exceptions can be made)
[ ] Not duplicating the functionality of existing packages.
[X] Not serving hacking, malware, phishing, spamming, spying, ddos functionality.
[X] I certify that I have read Termux Packaging Policy and understand that my request will be denied if it is found lacking.
Additional information

No response
TomJo2000 commented 1 month ago
We do not usually package Python modules unless they are widely used, and specifically require patching to work on Termux. Both of which are in question for this package.
What would help here would be:
Does this module require specific patches to work?
What is the range of applicability for this module?
shaheen-coder commented 1 month ago
Pytorch is popular
shaheen-coder commented 1 month ago
I can't torchtext through pip. Getting so many wheels build error
TomJo2000 commented 1 month ago
I can't torchtext through pip. Getting so many wheels build error
Could you provide those build errors? In full.
shaheen-coder commented 1 month ago
Can't build wheels for sentencepiece
Build log :
Building wheel for sentencepiece (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for sentencepiece (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [229 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-311
      creating build/lib.linux-aarch64-cpython-311/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-aarch64-cpython-311/sentencepiece
      copying src/sentencepiece/_version.py -> build/lib.linux-aarch64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-aarch64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-aarch64-cpython-311/sentencepiece
      running build_ext
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      CMake Deprecation Warning at CMakeLists.txt:15 (cmake_minimum_required):
        Compatibility with CMake < 3.5 will be removed from a future version of
        CMake.

        Update the VERSION argument <min> value or use a ...<max> suffix to tell
        CMake that the project does not need compatibility with older versions.

      -- VERSION: 0.2.0
      -- The C compiler identification is Clang 18.1.6
      -- The CXX compiler identification is Clang 18.1.6
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /data/data/com.termux/files/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /data/data/com.termux/files/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
      -- Looking for pthread_create in pthreads
      -- Looking for pthread_create in pthreads - not found
      -- Looking for pthread_create in pthread
      -- Looking for pthread_create in pthread - found
      -- Found Threads: TRUE
      -- Not Found TCMalloc: TCMALLOC_LIB-NOTFOUND
      -- Found atomic: /data/data/com.termux/files/usr/lib/libatomic.a
      -- Configuring done (4.5s)
      -- Generating done (0.1s)
      -- Build files have been written to: /data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/build
      [  1%] Building CXX object src/CMakeFiles/sentencepiece_train-static.dir/trainer_factory.cc.o
      [  3%] Building CXX object src/CMakeFiles/sentencepiece_train-static.dir/unicode_script.cc.o
      [  4%] Building CXX object src/CMakeFiles/sentencepiece_train-static.dir/trainer_interface.cc.o
      [  8%] Building CXX object src/CMakeFiles/sentencepiece_train-static.dir/word_model_trainer.cc.o
      [  6%] Building CXX object src/CMakeFiles/sentencepiece_train-static.dir/builder.cc.o
      [  9%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/arenastring.cc.o
      [ 11%] Building CXX object src/CMakeFiles/sentencepiece_train-static.dir/unigram_model_trainer.cc.o
      [ 12%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/arena.cc.o
      [ 14%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/bytestream.cc.o
      [ 16%] Building CXX object src/CMakeFiles/sentencepiece_train-static.dir/char_model_trainer.cc.o
      [ 17%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/coded_stream.cc.o
      [ 19%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/common.cc.o
      /data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/src/builder.cc:47:15: warning: unused variable 'kMaxUnicode' [-Wunused-const-variable]
         47 | constexpr int kMaxUnicode = 0x10FFFF;
            |               ^~~~~~~~~~~
      /data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/src/builder.cc:49:23: warning: unused variable 'kDefaultNormalizerName' [-Wunused-const-variable]
         49 | static constexpr char kDefaultNormalizerName[] = "nfkc";
            |                       ^~~~~~~~~~~~~~~~~~~~~~
      [ 20%] Building CXX object src/CMakeFiles/sentencepiece_train-static.dir/bpe_model_trainer.cc.o
      [ 22%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/extension_set.cc.o
      [ 24%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/generated_enum_util.cc.o
      [ 25%] Building CXX object src/CMakeFiles/sentencepiece_train-static.dir/sentencepiece_trainer.cc.o
      2 warnings generated.
      [ 27%] Building CXX object src/CMakeFiles/sentencepiece_train-static.dir/pretokenizer_for_training.cc.o
      [ 29%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/generated_message_table_driven_lite.cc.o
      [ 30%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/generated_message_util.cc.o
      [ 32%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/implicit_weak_message.cc.o
      /data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/src/sentencepiece_trainer.cc:207:16: warning: 'done' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
        207 |   virtual bool done() const { return iter_ == end_; }
            |                ^
      /data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/src/sentencepiece_trainer.h:52:16: note: overridden virtual function is here
         52 |   virtual bool done() const = 0;
            |                ^
      [ 33%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/int128.cc.o
      [ 35%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/io_win32.cc.o
      [ 37%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/message_lite.cc.o
      [ 38%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/parse_context.cc.o
      [ 40%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/repeated_field.cc.o
      [ 41%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/status.cc.o
      [ 43%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/statusor.cc.o
      [ 45%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/stringpiece.cc.o
      [ 46%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/stringprintf.cc.o
      1 warning generated.
      [ 48%] Linking CXX static library libsentencepiece_train.a
      [ 50%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/structurally_valid.cc.o
      [ 50%] Built target sentencepiece_train-static
      [ 51%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/strutil.cc.o
      [ 53%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/time.cc.o
      [ 54%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/wire_format_lite.cc.o
      [ 56%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/zero_copy_stream.cc.o
      [ 58%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/zero_copy_stream_impl.cc.o
      [ 59%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/protobuf-lite/zero_copy_stream_impl_lite.cc.o
      [ 61%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/builtin_pb/sentencepiece.pb.cc.o
      [ 62%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/builtin_pb/sentencepiece_model.pb.cc.o
      [ 64%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/bpe_model.cc.o
      [ 66%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/char_model.cc.o
      [ 67%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/error.cc.o
      [ 69%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/filesystem.cc.o
      [ 70%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/model_factory.cc.o
      [ 72%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/model_interface.cc.o
      [ 74%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/normalizer.cc.o
      [ 75%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/sentencepiece_processor.cc.o
      [ 77%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/unigram_model.cc.o
      [ 79%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/util.cc.o
      [ 80%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/word_model.cc.o
      [ 82%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/__/third_party/absl/flags/flag.cc.o
      [ 83%] Linking CXX static library libsentencepiece.a
      [ 83%] Built target sentencepiece-static
      [ 85%] Building CXX object src/CMakeFiles/spm_encode.dir/spm_encode_main.cc.o
      [ 87%] Building CXX object src/CMakeFiles/spm_decode.dir/spm_decode_main.cc.o
      [ 88%] Building CXX object src/CMakeFiles/spm_normalize.dir/spm_normalize_main.cc.o
      [ 90%] Building CXX object src/CMakeFiles/spm_export_vocab.dir/spm_export_vocab_main.cc.o
      [ 91%] Building CXX object src/CMakeFiles/spm_train.dir/spm_train_main.cc.o
      [ 93%] Linking CXX executable spm_export_vocab
      ld.lld: error: undefined symbol: __android_log_write
      >>> referenced by common.cc:150 (/data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/third_party/protobuf-lite/common.cc:150)
      >>>               common.cc.o:(google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)) in archive libsentencepiece.a
      >>> referenced by common.cc:158 (/data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/third_party/protobuf-lite/common.cc:158)
      >>>               common.cc.o:(google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)) in archive libsentencepiece.a
      c++: error: linker command failed with exit code 1 (use -v to see invocation)
      make[2]: *** [src/CMakeFiles/spm_export_vocab.dir/build.make:98: src/spm_export_vocab] Error 1
      make[1]: *** [CMakeFiles/Makefile2:286: src/CMakeFiles/spm_export_vocab.dir/all] Error 2
      make[1]: *** Waiting for unfinished jobs....
      [ 95%] Linking CXX executable spm_normalize
      ld.lld: error: undefined symbol: __android_log_write
      >>> referenced by common.cc:150 (/data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/third_party/protobuf-lite/common.cc:150)
      >>>               common.cc.o:(google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)) in archive libsentencepiece.a
      >>> referenced by common.cc:158 (/data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/third_party/protobuf-lite/common.cc:158)
      >>>               common.cc.o:(google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)) in archive libsentencepiece.a
      c++: error: linker command failed with exit code 1 (use -v to see invocation)
      make[2]: *** [src/CMakeFiles/spm_normalize.dir/build.make:100: src/spm_normalize] Error 1
      make[1]: *** [CMakeFiles/Makefile2:233: src/CMakeFiles/spm_normalize.dir/all] Error 2
      [ 96%] Linking CXX executable spm_train
      ld.lld: error: undefined symbol: __android_log_write
      >>> referenced by common.cc:150 (/data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/third_party/protobuf-lite/common.cc:150)
      >>>               common.cc.o:(google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)) in archive libsentencepiece.a
      >>> referenced by common.cc:158 (/data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/third_party/protobuf-lite/common.cc:158)
      >>>               common.cc.o:(google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)) in archive libsentencepiece.a
      [ 98%] Linking CXX executable spm_decode
      c++: error: linker command failed with exit code 1 (use -v to see invocation)
      make[2]: *** [src/CMakeFiles/spm_train.dir/build.make:100: src/spm_train] Error 1
      make[1]: *** [CMakeFiles/Makefile2:260: src/CMakeFiles/spm_train.dir/all] Error 2
      ld.lld: error: undefined symbol: __android_log_write
      >>> referenced by common.cc:150 (/data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/third_party/protobuf-lite/common.cc:150)
      >>>               common.cc.o:(google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)) in archive libsentencepiece.a
      >>> referenced by common.cc:158 (/data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/third_party/protobuf-lite/common.cc:158)
      >>>               common.cc.o:(google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)) in archive libsentencepiece.a
      c++: error: linker command failed with exit code 1 (use -v to see invocation)
      make[2]: *** [src/CMakeFiles/spm_decode.dir/build.make:98: src/spm_decode] Error 1
      make[1]: *** [CMakeFiles/Makefile2:206: src/CMakeFiles/spm_decode.dir/all] Error 2
      [100%] Linking CXX executable spm_encode
      ld.lld: error: undefined symbol: __android_log_write
      >>> referenced by common.cc:150 (/data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/third_party/protobuf-lite/common.cc:150)
      >>>               common.cc.o:(google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)) in archive libsentencepiece.a
      >>> referenced by common.cc:158 (/data/data/com.termux/files/usr/tmp/pip-install-dvkjpc_h/sentencepiece_0678cd4004894528a4f5fc5e7df54e93/sentencepiece/third_party/protobuf-lite/common.cc:158)
      >>>               common.cc.o:(google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)) in archive libsentencepiece.a
      c++: error: linker command failed with exit code 1 (use -v to see invocation)
      make[2]: *** [src/CMakeFiles/spm_encode.dir/build.make:98: src/spm_encode] Error 1
      make[1]: *** [CMakeFiles/Makefile2:180: src/CMakeFiles/spm_encode.dir/all] Error 2
      make: *** [Makefile:156: all] Error 2
      Traceback (most recent call last):
        File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 410, in build_wheel
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
          self.run_setup()
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 169, in <module>
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 968, in run_command
          super().run_command(command)
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/data/data/com.termux/files/usr/tmp/pip-build-env-3be_qtzy/normal/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 368, in run
termux / termux-packages
