shipping the model

[eubinecto]

Why?

In order to use the pipeline outside of this library. (e.g. idiom2vec & youtora)

The goal

distribute mip on a package.. so that I could do.. pip install mip_0.0.0?

But how?

there have been attempts to ship a model with a serialisable tokenizer.

• Shipping model with custom tokenizer and custom preprocessor

but obviously. my idiom_matcher is not serialisable. what a shame.

shipping with custom component?

you can ship the entire model package!

• https://spacy.io/api/cli#package

To-do's

• [x] use idiom_matcher.pkl, and ship with the binary. That's a better way. Make sure you sync the vocab. ~- [ ] ship the model with spacy_package. ~ : this is a pretty bad idea. just ship the component.
• [ ]. how do I distribute the project to pypi?

[eubinecto]

okay..well. It seems it's not just plug and play.

You have to install the library merge_idioms, before you install the model. hmm.. this is bad.. I want a single package which I can just ship.

• https://github.com/explosion/spaCy/issues/6352#issuecomment-722974017

what about this one?

• https://stackoverflow.com/questions/51412095/spacy-save-custom-pipeline

This person had exactly the same question that I had here.

By default, the init.py only includes a function to load your model – but you can also add custom functions to it or use it to add entries to spaCy's factories.

and..the problem with loading matcher from idiom_matcher.pkl

What if someone else want's to attach the component to their model? The current way of loading pkl and syncing the vocab will mess up the previous vocab.

[eubinecto]

a very good blog post on uploading the project on pip

• https://betterscientificsoftware.github.io/python-for-hpc/tutorials/python-pypi-packaging/#creating-a-python-package

[eubinecto]

to distribute your library to pypi, just run:

twine dist/*