Add module tests for most built-in modules

markgw / pimlico

The Pimlico Processing Toolkit

http://pimlico.readthedocs.org/

GNU Lesser General Public License v3.0

6 stars 1 forks source link

Add module tests for most built-in modules #7

Closed markgw closed 5 years ago

markgw commented 7 years ago

Massive project. Really need some fairly comprehensive unit tests for the core codebase before v1.

markgw commented 7 years ago

Unit tests are in https://github.com/markgw/pimlico/tree/master/src/test/python/pimlicotest and there's a script to run them, https://github.com/markgw/pimlico/blob/master/bin/all_tests.sh .

Currently there are two very simple tests. There will need to be many hundreds, so we've got a long way to go...

markgw commented 7 years ago

Commit 56c8e8c added a new feature for testing: test pipelines. This is intended to make it easy to build test suites for modules (and possibly datatypes), providing necessary data for input.

I intend to largely use this instead of unit tests. It would still be appropriate to create unit tests for core functionality, but realistically I'm not going to ever feel motivated enough to do this on a suitable scale. These test pipelines will effectively test a lot of core functionality as a by-product, since the modules will use a wide range of it, and they have the advantage that they also provide demo pipelines to show how to use the modules.

markgw commented 6 years ago

Now to be implemented as module tests instead of unit tests, see previous comment.

markgw commented 6 years ago

[x] caevo
[ ] candc
[ ] corenlp
[x] corpora.concat
[x] corpora.corpus_stats
[x] corpora.format
[x] corpora.list_filter
[x] corpora.split
[x] corpora.subset
[x] corpora.tar_filter
[ ] corpora.vocab_builder
[ ] corpora.vocab_mapper
[ ] embeddings.word2vec
[ ] features.term_feature_compiler
[ ] term_feature_matrix_builder
[ ] features.vocab_builder
[ ] features.vocab_mapper
[x] input.embeddings.fasttext
[ ] input.embeddings.fasttext_gensim
[x] input.text.raw_text_files
[ ] malt.parse
[ ] opennlp.coreference
[ ] opennlp.coreference_pipeline
[ ] opennlp.ner
[ ] opennlp.parse
[ ] opennlp.pos
[x] opennlp.tokenize
[ ] r.script
[ ] regex.annotated_text
[ ] sklearn.matrix_factorization
[ ] utility.alias
[ ] utility.copy_file
[ ] visualization.bar_chart
[ ] visualization.embeddings_plot

markgw commented 5 years ago

Most modules have tests now, so I'm closing this. I'm adding test pipelines as I update the remaining modules, so we should end up with all (or almost all) modules with tests.