Python with coccoc_tokenizer

[vietnamican]

In README file, i found this line: from CocCocTokenizer import PyTokenizer but how can i install CocCocTokenizer (I try copy and paste example and i got No module name: 'CocCocTokenizer' error)

[bachan]

You need to first install python bindings somewhere (see Installing part of the README). Here's an example of installing into your ~/.local/ folder (where pip --user will normally install stuff):

$ git clone https://github.com/coccoc/coccoc-tokenizer
$ cd coccoc-tokenizer && mdkir build && cd build
$ cmake -DBUILD_PYTHON=1 -DCMAKE_INSTALL_PREFIX=$HOME/.local ..
$ make install

After that you should be able to do

from CocCocTokenizer import PyTokenizer

We planned to add our module into pip repository, but I personally have no time to do that currently. Do you know how to add modules into pip?

[vietnamican]

@bachan Well, i know how to deploy a package to PyPI, but i dont know how to deploy a Cython package to PyPI, because of it's C++ dependency.

[dvquy13]

You need to first install python bindings somewhere (see Installing part of the README). Here's an example of installing into your ~/.local/ folder (where pip --user will normally install stuff):

$ git clone https://github.com/coccoc/coccoc-tokenizer
$ cd coccoc-tokenizer && mdkir build && cd build
$ cmake -DBUILD_PYTHON=1 -DCMAKE_INSTALL_PREFIX=$HOME/.local ..
$ make install

I would like to add that you should clone the repo to $HOME/.local directory to avoid permission error like this:

CMake Error at cmake_install.cmake:136 (file):
  file failed to open for writing (Permission denied):

    $HOME/open-source/coccoc-tokenizer/build/install_manifest.txt

Btw I use Ubuntu from Windows WSL.