search

daggaz / json-stream

Simple streaming JSON parser and encoder.
MIT License
122 stars 18 forks source link

Allow passing in custom tokenizer? #18

Closed smheidrich closed 2 years ago

smheidrich commented 2 years ago

To make this library a bit faster, I ported the tokenizer to Rust: https://github.com/smheidrich/py-json-stream-rs-tokenizer

Currently this works by monkeypatching the Rust tokenizer over json-stream's own tokenizer in loader and visitor, which is pretty hacky.

Do you think it would make sense to give load and visit an extra parameter that allows users to pass in a custom tokenizer? Then I could use that instead of monkeypatching.

daggaz commented 2 years ago

Hey, I'd love to see a PR with this?

daggaz commented 2 years ago

I've released a new version 1.4.0 with this to pypi, and added to the docs: https://github.com/daggaz/json-stream#custom-tokenizer

smheidrich commented 2 years ago

Perfect, thanks a lot! :+1: I've updated my package as well.