facelessuser
commented
1 year ago In the beginning, optimization wasn't the focus, simply automating spell-checking. I honestly didn't expect people to find it and for it to become as popular as it has started to become 🙂. I am aware of the impact related to performance as I do have a few projects that are quite large, and yes I notice the speed issue.
I should note that the built-in Python filters do not have to be used. If you are using Aspell, and the native Aspell filters are sufficient, you can just put Aspell into the correct mode and omit any of the Python filters this could potentially make things much faster. With that said, I often use the Python filters as PySpelling solves two issues I had: 1) I wanted to automate all the spell checking in CI, and 2) I felt at times the Aspell native filter modes were not sufficient for omitting certain content. At least for me.
PySpelling makes setting up automated spelling easier and in addition, allows you to use custom Python filters, but yeah, the generic interface and Python filtering process can be slower than the optimized Aspell filters written in C.
I imagine there are a few areas of possible improvement. One area is with how many times encoding and decoding of Unicode happens.
In general, we pass bytes content to the spell checkers. Content is read from the files as bytes content as well. Filters processing content for the first time need to process bytes content because we don't know what the encoding is yet, and encoding can be specified in headers for a given file type in a special way. For simplicity, all filters just take bytes buffers, but not all filters require bytes inputs for parsing the data once encoding is known.
Some filters that can handle Unicode fine and could be made Unicode aware, and what I mean by this is skip handling the buffer as bytes (once encoding is known). We could then keep buffers as Unicode (for most of the time). Filters would have to advertise they are Unicode ready and have an entry point for Unicode processing (assuming the encoding has already been identified by whatever the first filter was). They of course would need to return content as Unicode as well. I imagine this could reduce the encoding/decoding we do internally and give a speed boost. It does add complexity though, but it could have a noticeable improvement.
Another area that may cause overhead is the fact that PySpelling provides specific context for specific blocks. This may give you the function under which a docstring is found, or the HTML element under which some text is found. As we are often filtering out blocks of content, you lose such information as line numbers and such, so we provide what context we can, but in order to do this, each block is processed separately. More calls to the spell-checker means more overhead. We could add an option to just group all content for a given file at the end and make a single call. You'd lose context, but it'd be potentially faster.
The draw of writing this was that I could process content in advanced ways via 3rd party libraries without having to write low-level parsing for various file types. I could just use off-the-shelf libs. On the other side, you run into the issue where you don't have highly specialized, super-optimized filters, but they are flexible and powerful.
Anyway, to answer your question more directly, I am aware of the performance issues, I have considered making improvements, but it has also been lower in priority as not many people have yet complained about it, and it has been sufficient for my needs up until now. It does what it is supposed to do, but yes the speed on very large projects has been a mild annoyance for me.
benjamb
Hi there,
we, the Launchpad team at Canonical, have started using
pyspellingand we are very grateful for your work.Our project currently consists of about 50 pages, and will grow to several hundred.
With 50 pages spellchecking already takes almost a minute.
Do you see any chances to improve the speed? Or are there any intentions to do so?
Thank you!