sqlitebrowser / sqlitebrowser

Official home of the DB Browser for SQLite (DB4S) project. Previously known as "SQLite Database Browser" and "Database Browser for SQLite". Website at:
https://sqlitebrowser.org
Other
20.28k stars 2.09k forks source link

GUI frozen when browsing a large database file (100GB) #2877

Open jessie-github opened 2 years ago

jessie-github commented 2 years ago

Details for the issue

What did you do?

I opened a very large database file (100GB, one table, 50 columns, 1 000 000 rows) and I was trying to browse the data (using the Browse Data tab) to see the first rows and perhaps using some filter to look for some particular data. I set the preferences to have a small prefetch (300).

What did you expect to see?

I was expecting to be able to browse the data, at least have a look at the first rows and perhaps use some filtering to look for some particular data.

What did you see instead?

The number of rows appears at the bottom and then the GUI is completely frozen without any data appearing. After that I have to kill the application to get out.

What operating system are you using?

What is your DB4S version?

Did you also

Yes there is a similar issue there which was closed: https://github.com/sqlitebrowser/sqlitebrowser/issues/584 (but in my case the number of rows is appearing)

I also discussed the topic on Gitter before opening: https://gitter.im/sqlitebrowser/sqlitebrowser?at=616fe98ccd4972068b5a0df8

chrisjlocke commented 2 years ago

Just for more information, this was initially raised in the Gitter chat, @jessie-github provided this useful info...

Ok I looked at the Application Log and indeed to count the number of rows, it is using "SELECT COUNT()" and it is returning the result in 3 seconds (I am surprised !). The next query is "SELECT 'rowID', from table LIMIT 0; 253" and the GUI is frozen from there (the first lines does not appear). I tried executing this query outside and it is returning the result instantly. So there is something wrong around this last query but no means for me to debug this further (nothing in the Error Log as well).

and

Indeed and this what I am often doing as well. Sometimes I just want to browse through the first hundred of lines to see what is the content and other times I want to filter to have only have rows corresponding to the criteria. Problem here is that the GUI is frozen and I cannot enter any filter at the top of the column. I am using (abusing ?) it a bit like I would use Excel to browse/filter/display a table with a lot of data. For debugging it is really useful. Manipulating large CSV files (as input for other tools), it is really useful to be able to open the file (without loading the entire file in memory) and explore a bit the content of the file to see how to parse it (for instance to check that the date are of the standard YYYY-MM-DD instead of MM/DD/YYYY). Similarly for SQLite file, I wanted to have a peek without having to (manually) type a query to display the first hundreds of lines.

BiatuAutMiahn commented 4 months ago

Any updates on this?

chrisjlocke commented 4 months ago

What version / OS of DB4S are you using? I've just created a database with 64 million rows (albeit small rows) for a 1.5 GB database. Switching between tabs is instant with no great issues.

image

In preferences, and the 'database' tab, what value have you got in the 'prefetch block size'? This is how many records are pulled down for a 'page'.