We weren't able to simply swap in a simpler tokenizer because these tests rely on a pos tagger. Spacy is probably a sane default for that according to @DeNeutoy.
Version details for testing locally:
(allennlp-hub) brendanr.local ➜ allennlp-hub git:(fix_sniff) ✗ ipython
impPython 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 13:42:17)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.9.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import spacy
In [2]: spacy.version Out[2]: '2.2.2' (allennlp-hub) brendanr.local ➜ allennlp-hub git:(fix_sniff) ✗ python -m spacy validate ✔ Loaded compatibility table
====================== Installed models (spaCy v2.2.2) ====================== ℹ spaCy installation: /Users/brendanr/anaconda3/envs/allennlp-hub/lib/python3.6/site-packages/spacy
TYPE NAME MODEL VERSION package en-core-web-sm en_core_web_sm 2.2.5 ✔