Open bpatrik opened 3 years ago
Perhaps an option would be to use "better-sqlite3" in combination with TypeORM? https://typeorm.io/#/connection-options/better-sqlite3-connection-options
better-sqlite3 seems to perform much faster than sqlite3 and in combination with TypeORM it seems to be simply a matter of switching out the npm package and TypeORM connection option from sqlite3 to better-sqlite3.
I have just tested the idea and it is simply a matter of switching out the existing sqlite3 dependency for better-sqlite3 (tested version 7.4.4). Since TypeORM is already compatible with better-sqlite3, you can then simply change the driver from type: 'sqlite'
to type: 'better-sqlite3'
in SQLConnection.ts (Line 195).
The positive performance impact is massive, especially on my database with ~150.000 images.
better-sqlite3 is compatible to any existing sqlite database files, so it should really be as simple as that.
On another note, I'd also update TypeORM itself to a more recent version, but I haven't tested whether a TypeORM update breaks anything yet.
This sounds great! I would need to spend some time with it to double check everything works and it indeed ads a speed up tho the app.
I already have some benchmark (that code is only best-effort maintained): https://github.com/bpatrik/pigallery2/tree/master/benchmark
I would need to run that to compare the benefit.
compared better-sqlite3 with sqlite3 on my rp4, using my benchmark
Is seems that reading or writing a lot of records with better-sqlite3
is significantly faster, while the other operations are mostly equal (or slightly faster).
Here is the row data:
NOTE: if sqlite driver is not specified than it is unknown (some cases better-sqlite3, while some other times sqlite3). (sorry, I was lazy to remove it from the data)
Version: v1.9.1, built at: Tue Feb 22 2022 13:23:52 GMT+0000 (Coordinated Universal Time), git commit:b1f094c1501c9435c579ae242f35442ff96d9c5e System: Raspberry Pi 4 4G Model B, SandisK Mobile Ultra 32Gb CLass10, UHS-I, HDD: Western Digital Elements 1TB (WDBUZG0010BBK)
Gallery: directories: 31, photos: 2036, videos: 35, diskUsage : 22.08GB, persons : 1381, unique persons (faces): 25
Action | Sub action | Average Duration | Result |
---|---|---|---|
Scanning directory | 10360.5 ms | media: 698, directories: 0 | |
[SQlite=sqlite3] Scanning directory |
10628.2 ms | media: 698, directories: 0 | |
[SQlite=better-sqlite] Scanning directory |
11169.9 ms | media: 698, directories: 0 | |
Saving directory to DB | 4264.7 ms | - | |
[SQlite=sqlite3] Saving directory to DB |
3655.3 ms | - | |
[SQlite=better-sqlite] Saving directory to DB |
1988.2 ms | - | |
List directory | 417.8 ms | media: 698, directories: 0 | |
Authenticate | 0.1 ms | - | |
Normalize path param | 0.0 ms | - | |
Authorise path | 0.0 ms | - | |
Inject gallery version | 3.8 ms | - | |
List directory | 334.3 ms | media: 698, directories: 0 | |
Add thumbnail information | 60.2 ms | media: 698, directories: 0 | |
Clean up gallery results | 18.8 ms | media: 698, directories: 0 | |
Render result | 0.0 ms | media: 698, directories: 0 | |
[SQlite=sqlite3] List directory |
416.9 ms | media: 698, directories: 0 | |
Authenticate | 0.1 ms | - | |
Normalize path param | 0.0 ms | - | |
Authorise path | 0.0 ms | - | |
Inject gallery version | 3.8 ms | - | |
List directory | 338.8 ms | media: 698, directories: 0 | |
Add thumbnail information | 55.4 ms | media: 698, directories: 0 | |
Clean up gallery results | 18.5 ms | media: 698, directories: 0 | |
Render result | 0.0 ms | media: 698, directories: 0 | |
[SQlite=better-sqlite] List directory |
325.7 ms | media: 698, directories: 0 | |
Authenticate | 0.1 ms | - | |
Normalize path param | 0.0 ms | - | |
Authorise path | 0.0 ms | - | |
Inject gallery version | 2.7 ms | - | |
List directory | 249.5 ms | media: 698, directories: 0 | |
Add thumbnail information | 55.7 ms | media: 698, directories: 0 | |
Clean up gallery results | 17.4 ms | media: 698, directories: 0 | |
Render result | 0.0 ms | media: 698, directories: 0 | |
Listing Faces | 44.7 ms | items: 25 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 4.7 ms | - | |
List persons | 36.4 ms | items: 25 | |
Add thumbnail info for persons | 1.6 ms | items: 25 | |
Clean up person results | 1.7 ms | items: 25 | |
Render result | 0.0 ms | items: 25 | |
[SQlite=sqlite3] Listing Faces |
46.5 ms | items: 25 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 5.4 ms | - | |
List persons | 37.4 ms | items: 25 | |
Add thumbnail info for persons | 1.7 ms | items: 25 | |
Clean up person results | 1.7 ms | items: 25 | |
Render result | 0.0 ms | items: 25 | |
[SQlite=better-sqlite] Listing Faces |
7.5 ms | - | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 2.0 ms | - | |
List persons | 5.0 ms | - | |
Add thumbnail info for persons | 0.0 ms | - | |
Clean up person results | 0.1 ms | - | |
Render result | 0.0 ms | - | |
Searching for a |
933.4 ms | media: 2001, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 708.5 ms | media: 2001, directories: 0 | |
Add thumbnail information | 174.6 ms | media: 2001, directories: 0 | |
Clean up gallery results | 49.9 ms | media: 2001, directories: 0 | |
Render result | 0.0 ms | media: 2001, directories: 0 | |
[SQlite=sqlite3] Searching for a |
933.5 ms | media: 2001, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 711.5 ms | media: 2001, directories: 0 | |
Add thumbnail information | 172.2 ms | media: 2001, directories: 0 | |
Clean up gallery results | 49.2 ms | media: 2001, directories: 0 | |
Render result | 0.0 ms | media: 2001, directories: 0 | |
[SQlite=better-sqlite] Searching for a |
933.2 ms | media: 2001, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 705.7 ms | media: 2001, directories: 0 | |
Add thumbnail information | 177.4 ms | media: 2001, directories: 0 | |
Clean up gallery results | 49.6 ms | media: 2001, directories: 0 | |
Render result | 0.0 ms | media: 2001, directories: 0 | |
Searching for caption:a |
12.1 ms | media: 0, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 11.6 ms | media: 0, directories: 0 | |
Add thumbnail information | 0.1 ms | media: 0, directories: 0 | |
Clean up gallery results | 0.0 ms | media: 0, directories: 0 | |
Render result | 0.0 ms | media: 0, directories: 0 | |
[SQlite=sqlite3] Searching for caption:a |
11.4 ms | media: 0, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 11.0 ms | media: 0, directories: 0 | |
Add thumbnail information | 0.1 ms | media: 0, directories: 0 | |
Clean up gallery results | 0.0 ms | media: 0, directories: 0 | |
Render result | 0.0 ms | media: 0, directories: 0 | |
[SQlite=better-sqlite] Searching for caption:a |
11.4 ms | media: 0, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 10.9 ms | media: 0, directories: 0 | |
Add thumbnail information | 0.1 ms | media: 0, directories: 0 | |
Clean up gallery results | 0.0 ms | media: 0, directories: 0 | |
Render result | 0.0 ms | media: 0, directories: 0 | |
Searching for directory:a |
792.4 ms | media: 1705, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 603.9 ms | media: 1705, directories: 0 | |
Add thumbnail information | 146.6 ms | media: 1705, directories: 0 | |
Clean up gallery results | 41.4 ms | media: 1705, directories: 0 | |
Render result | 0.0 ms | media: 1705, directories: 0 | |
[SQlite=sqlite3] Searching for directory:a |
806.8 ms | media: 1705, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 618.6 ms | media: 1705, directories: 0 | |
Add thumbnail information | 146.7 ms | media: 1705, directories: 0 | |
Clean up gallery results | 41.0 ms | media: 1705, directories: 0 | |
Render result | 0.0 ms | media: 1705, directories: 0 | |
[SQlite=better-sqlite] Searching for directory:a |
792.1 ms | media: 1705, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 599.9 ms | media: 1705, directories: 0 | |
Add thumbnail information | 150.5 ms | media: 1705, directories: 0 | |
Clean up gallery results | 41.2 ms | media: 1705, directories: 0 | |
Render result | 0.0 ms | media: 1705, directories: 0 | |
Searching for file-name:a |
48.2 ms | media: 79, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 39.6 ms | media: 79, directories: 0 | |
Add thumbnail information | 6.4 ms | media: 79, directories: 0 | |
Clean up gallery results | 1.8 ms | media: 79, directories: 0 | |
Render result | 0.0 ms | media: 79, directories: 0 | |
[SQlite=sqlite3] Searching for file-name:a |
58.8 ms | media: 79, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 49.3 ms | media: 79, directories: 0 | |
Add thumbnail information | 7.1 ms | media: 79, directories: 0 | |
Clean up gallery results | 2.0 ms | media: 79, directories: 0 | |
Render result | 0.0 ms | media: 79, directories: 0 | |
[SQlite=better-sqlite] Searching for file-name:a |
47.8 ms | media: 79, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 39.2 ms | media: 79, directories: 0 | |
Add thumbnail information | 6.4 ms | media: 79, directories: 0 | |
Clean up gallery results | 1.8 ms | media: 79, directories: 0 | |
Render result | 0.0 ms | media: 79, directories: 0 | |
Searching for keyword:a |
720.7 ms | media: 1536, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 546.6 ms | media: 1536, directories: 0 | |
Add thumbnail information | 137.1 ms | media: 1536, directories: 0 | |
Clean up gallery results | 36.4 ms | media: 1536, directories: 0 | |
Render result | 0.0 ms | media: 1536, directories: 0 | |
[SQlite=sqlite3] Searching for keyword:a |
707.6 ms | media: 1536, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 539.8 ms | media: 1536, directories: 0 | |
Add thumbnail information | 130.6 ms | media: 1536, directories: 0 | |
Clean up gallery results | 36.7 ms | media: 1536, directories: 0 | |
Render result | 0.0 ms | media: 1536, directories: 0 | |
[SQlite=better-sqlite] Searching for keyword:a |
732.8 ms | media: 1536, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 563.4 ms | media: 1536, directories: 0 | |
Add thumbnail information | 130.7 ms | media: 1536, directories: 0 | |
Clean up gallery results | 38.2 ms | media: 1536, directories: 0 | |
Render result | 0.0 ms | media: 1536, directories: 0 | |
Searching for person:a |
402.3 ms | media: 825, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 310.6 ms | media: 825, directories: 0 | |
Add thumbnail information | 68.4 ms | media: 825, directories: 0 | |
Clean up gallery results | 22.8 ms | media: 825, directories: 0 | |
Render result | 0.0 ms | media: 825, directories: 0 | |
[SQlite=sqlite3] Searching for person:a |
403.7 ms | media: 825, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.2 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 311.9 ms | media: 825, directories: 0 | |
Add thumbnail information | 68.2 ms | media: 825, directories: 0 | |
Clean up gallery results | 23.1 ms | media: 825, directories: 0 | |
Render result | 0.0 ms | media: 825, directories: 0 | |
[SQlite=better-sqlite] Searching for person:a |
410.3 ms | media: 825, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 317.9 ms | media: 825, directories: 0 | |
Add thumbnail information | 68.2 ms | media: 825, directories: 0 | |
Clean up gallery results | 23.7 ms | media: 825, directories: 0 | |
Render result | 0.0 ms | media: 825, directories: 0 | |
Searching for position:a |
537.7 ms | media: 1133, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 414.3 ms | media: 1133, directories: 0 | |
Add thumbnail information | 96.7 ms | media: 1133, directories: 0 | |
Clean up gallery results | 26.3 ms | media: 1133, directories: 0 | |
Render result | 0.0 ms | media: 1133, directories: 0 | |
[SQlite=sqlite3] Searching for position:a |
551.7 ms | media: 1133, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.4 ms | - | |
Search | 430.7 ms | media: 1133, directories: 0 | |
Add thumbnail information | 94.9 ms | media: 1133, directories: 0 | |
Clean up gallery results | 25.4 ms | media: 1133, directories: 0 | |
Render result | 0.0 ms | media: 1133, directories: 0 | |
[SQlite=better-sqlite] Searching for position:a |
543.1 ms | media: 1133, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 416.8 ms | media: 1133, directories: 0 | |
Add thumbnail information | 98.5 ms | media: 1133, directories: 0 | |
Clean up gallery results | 27.3 ms | media: 1133, directories: 0 | |
Render result | 0.0 ms | media: 1133, directories: 0 | |
Searching for . |
945.6 ms | media: 2001, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 711.9 ms | media: 2001, directories: 0 | |
Add thumbnail information | 179.4 ms | media: 2001, directories: 0 | |
Clean up gallery results | 53.8 ms | media: 2001, directories: 0 | |
Render result | 0.0 ms | media: 2001, directories: 0 | |
[SQlite=sqlite3] Searching for . |
941.7 ms | media: 2001, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 710.4 ms | media: 2001, directories: 0 | |
Add thumbnail information | 174.4 ms | media: 2001, directories: 0 | |
Clean up gallery results | 56.4 ms | media: 2001, directories: 0 | |
Render result | 0.0 ms | media: 2001, directories: 0 | |
[SQlite=better-sqlite] Searching for . |
948.6 ms | media: 2001, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 714.1 ms | media: 2001, directories: 0 | |
Add thumbnail information | 176.9 ms | media: 2001, directories: 0 | |
Clean up gallery results | 57.1 ms | media: 2001, directories: 0 | |
Render result | 0.0 ms | media: 2001, directories: 0 | |
Searching for <Most common name> |
141.9 ms | media: 262, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 112.6 ms | media: 262, directories: 0 | |
Add thumbnail information | 21.6 ms | media: 262, directories: 0 | |
Clean up gallery results | 7.3 ms | media: 262, directories: 0 | |
Render result | 0.0 ms | media: 262, directories: 0 | |
[SQlite=sqlite3] Searching for <Most common name> |
142.0 ms | media: 262, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 112.8 ms | media: 262, directories: 0 | |
Add thumbnail information | 21.9 ms | media: 262, directories: 0 | |
Clean up gallery results | 6.9 ms | media: 262, directories: 0 | |
Render result | 0.0 ms | media: 262, directories: 0 | |
[SQlite=better-sqlite] Searching for <Most common name> |
143.1 ms | media: 262, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 114.8 ms | media: 262, directories: 0 | |
Add thumbnail information | 21.0 ms | media: 262, directories: 0 | |
Clean up gallery results | 6.9 ms | media: 262, directories: 0 | |
Render result | 0.0 ms | media: 262, directories: 0 | |
Searching for <Most AND second common names> |
30.5 ms | media: 20, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 27.7 ms | media: 20, directories: 0 | |
Add thumbnail information | 1.8 ms | media: 20, directories: 0 | |
Clean up gallery results | 0.7 ms | media: 20, directories: 0 | |
Render result | 0.0 ms | media: 20, directories: 0 | |
[SQlite=sqlite3] Searching for <Most AND second common names> |
32.8 ms | media: 20, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 29.8 ms | media: 20, directories: 0 | |
Add thumbnail information | 1.9 ms | media: 20, directories: 0 | |
Clean up gallery results | 0.7 ms | media: 20, directories: 0 | |
Render result | 0.0 ms | media: 20, directories: 0 | |
[SQlite=better-sqlite] Searching for <Most AND second common names> |
28.7 ms | media: 20, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 26.0 ms | media: 20, directories: 0 | |
Add thumbnail information | 1.7 ms | media: 20, directories: 0 | |
Clean up gallery results | 0.6 ms | media: 20, directories: 0 | |
Render result | 0.0 ms | media: 20, directories: 0 | |
Searching for <Most OR second common names> |
249.1 ms | media: 448, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 199.0 ms | media: 448, directories: 0 | |
Add thumbnail information | 37.7 ms | media: 448, directories: 0 | |
Clean up gallery results | 12.0 ms | media: 448, directories: 0 | |
Render result | 0.0 ms | media: 448, directories: 0 | |
[SQlite=sqlite3] Searching for <Most OR second common names> |
230.5 ms | media: 448, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 183.4 ms | media: 448, directories: 0 | |
Add thumbnail information | 35.6 ms | media: 448, directories: 0 | |
Clean up gallery results | 11.1 ms | media: 448, directories: 0 | |
Render result | 0.0 ms | media: 448, directories: 0 | |
[SQlite=better-sqlite] Searching for <Most OR second common names> |
251.6 ms | media: 448, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 202.2 ms | media: 448, directories: 0 | |
Add thumbnail information | 36.7 ms | media: 448, directories: 0 | |
Clean up gallery results | 12.3 ms | media: 448, directories: 0 | |
Render result | 0.0 ms | media: 448, directories: 0 | |
Searching for <Contain at least 2 out of all names> |
452.9 ms | media: 323, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 416.2 ms | media: 323, directories: 0 | |
Add thumbnail information | 26.7 ms | media: 323, directories: 0 | |
Clean up gallery results | 9.5 ms | media: 323, directories: 0 | |
Render result | 0.0 ms | media: 323, directories: 0 | |
[SQlite=sqlite3] Searching for <Contain at least 2 out of all names> |
469.4 ms | media: 323, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 433.0 ms | media: 323, directories: 0 | |
Add thumbnail information | 26.7 ms | media: 323, directories: 0 | |
Clean up gallery results | 9.3 ms | media: 323, directories: 0 | |
Render result | 0.0 ms | media: 323, directories: 0 | |
[SQlite=better-sqlite] Searching for <Contain at least 2 out of all names> |
449.2 ms | media: 323, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 410.9 ms | media: 323, directories: 0 | |
Add thumbnail information | 28.2 ms | media: 323, directories: 0 | |
Clean up gallery results | 9.6 ms | media: 323, directories: 0 | |
Render result | 0.0 ms | media: 323, directories: 0 | |
Auto complete for a |
17.3 ms | items: 24 | |
Authenticate | 0.2 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Autocomplete | 16.8 ms | items: 24 | |
Render result | 0.0 ms | items: 24 | |
[SQlite=sqlite3] Auto complete for a |
20.2 ms | items: 24 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Autocomplete | 19.8 ms | items: 24 | |
Render result | 0.0 ms | items: 24 | |
[SQlite=better-sqlite] Auto complete for a |
16.7 ms | items: 24 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Autocomplete | 16.4 ms | items: 24 | |
Render result | 0.0 ms | items: 24 |
*Measurements run 20 times and an average was calculated.
run for : 1720750.0ms
The previous benchmark result was corrupted so I had to rerun it.
There is a 10-12% increase with better-sqlite3
for every scenarios.
Version: v1.9.1, built at: Tue Feb 22 2022 23:16:30 GMT+0000 (Coordinated Universal Time), git commit:b4c64d12472bb618787168009eae103d59487ded System: Raspberry Pi 4 4G Model B, SandisK Mobile Ultra 32Gb CLass10, UHS-I, HDD: Western Digital Elements 1TB (WDBUZG0010BBK)
Gallery: directories: 31, photos: 2036, videos: 35, diskUsage : 22.08GB, persons : 1381, unique persons (faces): 25
Action | Sub action | Average Duration | Result |
---|---|---|---|
Scanning directory | 10184.3 ms | media: 698, directories: 0 | |
[SQlite=better-sqlite] Scanning directory |
10177.5 ms | media: 698, directories: 0 | |
Saving directory to DB | 4095.6 ms | - | |
[SQlite=better-sqlite] Saving directory to DB |
1963.8 ms | - | |
List directory | 411.9 ms | media: 698, directories: 0 | |
Authenticate | 0.1 ms | - | |
Normalize path param | 0.0 ms | - | |
Authorise path | 0.0 ms | - | |
Inject gallery version | 3.9 ms | - | |
List directory | 328.3 ms | media: 698, directories: 0 | |
Add thumbnail information | 58.8 ms | media: 698, directories: 0 | |
Clean up gallery results | 20.5 ms | media: 698, directories: 0 | |
Render result | 0.0 ms | media: 698, directories: 0 | |
[SQlite=better-sqlite] List directory |
326.5 ms | media: 698, directories: 0 | |
Authenticate | 0.1 ms | - | |
Normalize path param | 0.0 ms | - | |
Authorise path | 0.0 ms | - | |
Inject gallery version | 2.7 ms | - | |
List directory | 248.1 ms | media: 698, directories: 0 | |
Add thumbnail information | 57.1 ms | media: 698, directories: 0 | |
Clean up gallery results | 18.1 ms | media: 698, directories: 0 | |
Render result | 0.0 ms | media: 698, directories: 0 | |
Listing Faces | 43.5 ms | items: 25 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 3.4 ms | - | |
List persons | 36.5 ms | items: 25 | |
Add thumbnail info for persons | 1.6 ms | items: 25 | |
Clean up person results | 1.7 ms | items: 25 | |
Render result | 0.0 ms | items: 25 | |
[SQlite=better-sqlite] Listing Faces |
38.6 ms | items: 25 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 2.4 ms | - | |
List persons | 31.1 ms | items: 25 | |
Add thumbnail info for persons | 2.3 ms | items: 25 | |
Clean up person results | 2.5 ms | items: 25 | |
Render result | 0.0 ms | items: 25 | |
Searching for a |
937.2 ms | media: 2001, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 716.2 ms | media: 2001, directories: 0 | |
Add thumbnail information | 171.1 ms | media: 2001, directories: 0 | |
Clean up gallery results | 49.3 ms | media: 2001, directories: 0 | |
Render result | 0.0 ms | media: 2001, directories: 0 | |
[SQlite=better-sqlite] Searching for a |
818.5 ms | media: 2001, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 604.7 ms | media: 2001, directories: 0 | |
Add thumbnail information | 165.0 ms | media: 2001, directories: 0 | |
Clean up gallery results | 48.2 ms | media: 2001, directories: 0 | |
Render result | 0.0 ms | media: 2001, directories: 0 | |
Searching for caption:a |
10.8 ms | media: 0, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 10.4 ms | media: 0, directories: 0 | |
Add thumbnail information | 0.1 ms | media: 0, directories: 0 | |
Clean up gallery results | 0.0 ms | media: 0, directories: 0 | |
Render result | 0.0 ms | media: 0, directories: 0 | |
[SQlite=better-sqlite] Searching for caption:a |
9.5 ms | media: 0, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 9.0 ms | media: 0, directories: 0 | |
Add thumbnail information | 0.1 ms | media: 0, directories: 0 | |
Clean up gallery results | 0.0 ms | media: 0, directories: 0 | |
Render result | 0.0 ms | media: 0, directories: 0 | |
Searching for directory:a |
803.2 ms | media: 1705, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 610.4 ms | media: 1705, directories: 0 | |
Add thumbnail information | 151.6 ms | media: 1705, directories: 0 | |
Clean up gallery results | 40.6 ms | media: 1705, directories: 0 | |
Render result | 0.0 ms | media: 1705, directories: 0 | |
[SQlite=better-sqlite] Searching for directory:a |
689.0 ms | media: 1705, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 502.3 ms | media: 1705, directories: 0 | |
Add thumbnail information | 145.9 ms | media: 1705, directories: 0 | |
Clean up gallery results | 40.2 ms | media: 1705, directories: 0 | |
Render result | 0.0 ms | media: 1705, directories: 0 | |
Searching for file-name:a |
47.0 ms | media: 79, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 38.4 ms | media: 79, directories: 0 | |
Add thumbnail information | 6.3 ms | media: 79, directories: 0 | |
Clean up gallery results | 1.9 ms | media: 79, directories: 0 | |
Render result | 0.0 ms | media: 79, directories: 0 | |
[SQlite=better-sqlite] Searching for file-name:a |
43.6 ms | media: 79, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 33.9 ms | media: 79, directories: 0 | |
Add thumbnail information | 7.3 ms | media: 79, directories: 0 | |
Clean up gallery results | 2.0 ms | media: 79, directories: 0 | |
Render result | 0.0 ms | media: 79, directories: 0 | |
Searching for keyword:a |
723.3 ms | media: 1536, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 548.7 ms | media: 1536, directories: 0 | |
Add thumbnail information | 137.8 ms | media: 1536, directories: 0 | |
Clean up gallery results | 36.2 ms | media: 1536, directories: 0 | |
Render result | 0.0 ms | media: 1536, directories: 0 | |
[SQlite=better-sqlite] Searching for keyword:a |
622.2 ms | media: 1536, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 455.3 ms | media: 1536, directories: 0 | |
Add thumbnail information | 131.9 ms | media: 1536, directories: 0 | |
Clean up gallery results | 34.5 ms | media: 1536, directories: 0 | |
Render result | 0.0 ms | media: 1536, directories: 0 | |
Searching for person:a |
409.4 ms | media: 825, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 315.9 ms | media: 825, directories: 0 | |
Add thumbnail information | 71.7 ms | media: 825, directories: 0 | |
Clean up gallery results | 21.3 ms | media: 825, directories: 0 | |
Render result | 0.0 ms | media: 825, directories: 0 | |
[SQlite=better-sqlite] Searching for person:a |
355.6 ms | media: 825, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 266.1 ms | media: 825, directories: 0 | |
Add thumbnail information | 67.8 ms | media: 825, directories: 0 | |
Clean up gallery results | 21.1 ms | media: 825, directories: 0 | |
Render result | 0.0 ms | media: 825, directories: 0 | |
Searching for position:a |
547.7 ms | media: 1133, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 419.1 ms | media: 1133, directories: 0 | |
Add thumbnail information | 96.5 ms | media: 1133, directories: 0 | |
Clean up gallery results | 31.6 ms | media: 1133, directories: 0 | |
Render result | 0.0 ms | media: 1133, directories: 0 | |
[SQlite=better-sqlite] Searching for position:a |
486.1 ms | media: 1133, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 365.4 ms | media: 1133, directories: 0 | |
Add thumbnail information | 93.8 ms | media: 1133, directories: 0 | |
Clean up gallery results | 26.4 ms | media: 1133, directories: 0 | |
Render result | 0.0 ms | media: 1133, directories: 0 | |
Searching for . |
955.6 ms | media: 2001, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 732.7 ms | media: 2001, directories: 0 | |
Add thumbnail information | 174.5 ms | media: 2001, directories: 0 | |
Clean up gallery results | 47.8 ms | media: 2001, directories: 0 | |
Render result | 0.0 ms | media: 2001, directories: 0 | |
[SQlite=better-sqlite] Searching for . |
809.4 ms | media: 2001, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.5 ms | - | |
Search | 598.4 ms | media: 2001, directories: 0 | |
Add thumbnail information | 162.8 ms | media: 2001, directories: 0 | |
Clean up gallery results | 47.3 ms | media: 2001, directories: 0 | |
Render result | 0.0 ms | media: 2001, directories: 0 | |
Searching for <Most common name> |
156.5 ms | media: 262, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 122.7 ms | media: 262, directories: 0 | |
Add thumbnail information | 23.8 ms | media: 262, directories: 0 | |
Clean up gallery results | 9.5 ms | media: 262, directories: 0 | |
Render result | 0.0 ms | media: 262, directories: 0 | |
[SQlite=better-sqlite] Searching for <Most common name> |
122.5 ms | media: 262, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 92.5 ms | media: 262, directories: 0 | |
Add thumbnail information | 22.6 ms | media: 262, directories: 0 | |
Clean up gallery results | 7.0 ms | media: 262, directories: 0 | |
Render result | 0.0 ms | media: 262, directories: 0 | |
Searching for <Most AND second common names> |
28.9 ms | media: 20, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 26.2 ms | media: 20, directories: 0 | |
Add thumbnail information | 1.7 ms | media: 20, directories: 0 | |
Clean up gallery results | 0.6 ms | media: 20, directories: 0 | |
Render result | 0.0 ms | media: 20, directories: 0 | |
[SQlite=better-sqlite] Searching for <Most AND second common names> |
27.4 ms | media: 20, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Search | 24.5 ms | media: 20, directories: 0 | |
Add thumbnail information | 1.8 ms | media: 20, directories: 0 | |
Clean up gallery results | 0.7 ms | media: 20, directories: 0 | |
Render result | 0.0 ms | media: 20, directories: 0 | |
Searching for <Most OR second common names> |
236.0 ms | media: 448, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 187.7 ms | media: 448, directories: 0 | |
Add thumbnail information | 36.0 ms | media: 448, directories: 0 | |
Clean up gallery results | 11.6 ms | media: 448, directories: 0 | |
Render result | 0.1 ms | media: 448, directories: 0 | |
[SQlite=better-sqlite] Searching for <Most OR second common names> |
210.5 ms | media: 448, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 159.8 ms | media: 448, directories: 0 | |
Add thumbnail information | 38.1 ms | media: 448, directories: 0 | |
Clean up gallery results | 12.1 ms | media: 448, directories: 0 | |
Render result | 0.0 ms | media: 448, directories: 0 | |
Searching for <Contain at least 2 out of all names> |
461.2 ms | media: 323, directories: 0 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 421.6 ms | media: 323, directories: 0 | |
Add thumbnail information | 27.5 ms | media: 323, directories: 0 | |
Clean up gallery results | 11.6 ms | media: 323, directories: 0 | |
Render result | 0.0 ms | media: 323, directories: 0 | |
[SQlite=better-sqlite] Searching for <Contain at least 2 out of all names> |
382.0 ms | media: 323, directories: 0 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Search | 344.4 ms | media: 323, directories: 0 | |
Add thumbnail information | 27.7 ms | media: 323, directories: 0 | |
Clean up gallery results | 9.4 ms | media: 323, directories: 0 | |
Render result | 0.0 ms | media: 323, directories: 0 | |
Auto complete for a |
19.3 ms | items: 24 | |
Authenticate | 0.1 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.2 ms | - | |
Autocomplete | 18.9 ms | items: 24 | |
Render result | 0.0 ms | items: 24 | |
[SQlite=better-sqlite] Auto complete for a |
13.5 ms | items: 24 | |
Authenticate | 0.0 ms | - | |
Authorise | 0.0 ms | - | |
Inject gallery version | 0.1 ms | - | |
Autocomplete | 13.2 ms | items: 24 | |
Render result | 0.0 ms | items: 24 |
*Measurements run 20 times and an average was calculated.
run for : 2696203.0ms
Listing and searching is relatively slow currently, see: https://github.com/bpatrik/pigallery2/blob/master/benchmark/README.md.
Did some minimal local benchmarking and found that Issuing a
SELECT * FROM media_entry
takes <30ms from DB Browser from SQLite, while from code (even if I list raw results, not parsed JSON) it could take 300ms. This is without getting all the relations (directory, faces, etc) and doing anywhere
or sorting.A more realistic query:
In the DB app: 300ms, from nodejs: 7300ms (first run), 1000ms (following runs) (The long
faces
concatenation select is there as I figured that returning with less rows has 30-40% better performance.) I did the check on a DB with 21k photos, win 10, i7-6700HQ, with SSHD.There is clearly some significant performance loss around typerom.
There is already some harsh thread about it on reddit: TypeORM Sucks!! Something I wanted to talk about since long!
Currently I see two alternatives:
Blocking:
174