Ran against some errors with scaife-viewer/scaife-stack and doing local development where the existing "parallel tokenization" pipeline can hit weird exceptions that cause the whole pipeline to fail.
Would like to circle back around on some performance improvements for the tokenizer in the future, but for now, stability is key, and running the pipeline in serial provides that.
Ran against some errors with scaife-viewer/scaife-stack and doing local development where the existing "parallel tokenization" pipeline can hit weird exceptions that cause the whole pipeline to fail.
Would like to circle back around on some performance improvements for the tokenizer in the future, but for now, stability is key, and running the pipeline in serial provides that.