search

codota / tabnine-sublime

Tabnine Autocomplete AI: JavaScript, Python, TypeScript, PHP, C/C++, HTML/CSS, Go, Java, Ruby, C#, Rust, SQL, Bash, Kotlin, Julia, Lua, OCaml, Perl, Haskell, React
https://www.tabnine.com/install/sublime
MIT License
194 stars 36 forks source link

TabNine suggestions block Sublime for multiple seconds (on popup and selection) #113

Open kakra opened 2 years ago

kakra commented 2 years ago

please complete the following information:

Issue Details:

During auto-complete (just when the popup is supposed to show up), Sublime often blocks short amounts of time, sometimes several seconds. Looking at the process monitor, you can see that TabNine hangs in disk state and spikes IO usage. This blocking makes working and typing in code unusable. Selecting a suggested TabNine entry also blocks the editor for up to multiple seconds sometimes. This blocking happens like every 3-4 keypresses.

Background: metrics.db is stored on a btrfs partition which uses copy-on-write. If I recreate .config/TabNine with chattr +C and copy all contents from a backup, this file inherits the nocow-attribute. With this, Sublime feels a little bit smoother but it's still not very good.

TabNine should probably improve its IO patterns, and be copy-on-write friendly.

The system has 24 GB of RAM, and 11+ GB of that a available memory (around 10 GB of that is page cache). So this is not a low memory condition.

gz#6848

boaz-codota commented 2 years ago

Hey @kakra

Thank you for the awesome breakdown of the issue. We indeed not friendly to copy-on-write situations, and we will look into improving that.

Do you have any suggestions as to how to improve this? Or must we sacrifice consistency and batch write the state?

kakra commented 2 years ago

I think SQlite has a solution for this by using write-ahead logging (that's the WAL files), here's some starting point on the topic: https://wiki.tnonline.net/w/Blog/SQLite_Performance_on_Btrfs

Maybe the database files of TabNine should use something similar (I think you already use SQlite).

I've already done that for other SQlite databases and it can be done without knowledge for that application using the steps described in the link, section "How to enable SQLite WAL" (bottom of the page).

kakra commented 2 years ago

If possible, try to avoid fsync patterns for synchronization - fsync is quite horrible on btrfs performance-wise. Btrfs doesn't need it being a cow file system. But I'm not sure how to handle that, maybe through some env var like TABNINE_NO_FSYNC=1 or something that advanced users could set manually in their profile.

kakra commented 2 years ago

Also, avoid in-place updates of blocks in files and prefer appending data to some sort of log file which you could merge at process start or regular intervals back into the data file (similar to WAL).

kakra commented 2 years ago

I'm not sure if SQlite has file access modes which use fsync() - if they do, I'd research if that is really needed with WAL, and if it isn't, you may want to explicitly disable that access mode. Btrfs really doesn't like fsync because it flushes internal trees and transactions.