search

stashapp / stash

An organizer for your porn, written in Go. Documentation: https://docs.stashapp.cc
https://stashapp.cc/
GNU Affero General Public License v3.0
9.12k stars 790 forks source link

Files refactor bug reports #2737

Closed WithoutPants closed 2 years ago

WithoutPants commented 2 years ago

This is intended as a catch-all issue to report bugs with the files-refactor alpha branch.

Known issues are as follows:

Other notable changes:

Please report issues with the files-refactor build here.

7dJx1qP commented 2 years ago

I got about a dozen of these errors while running my first full library scan post-migration:

level=error msg="updating files: executing `UPDATE `files` SET `basename`=?,`created_at`=?,`id`=?,`mod_time`=?,`parent_folder_id`=?,`size`=?,`updated_at`=?,`zip_file_id`=? WHERE (`files`.`id` = ?)` [[XXXXXXXXXX.avi 2022-08-04 09:36:28.7004594 -0400 -0400 672814 2022-06-30 09:30:11 -0400 -0400 8976 495341568 2022-08-04 09:36:28.7004594 -0400 -0400 <nil> 672814]]: database is locked"

And then a few of these errors:

level=error msg="error processing \"I:\\\\Sites\\\\XXXX\\\\images\\\\XXXX\\\\crop\\\\460x349\\\\nt\": creating folder \"I:\\\\Sites\\\\XXXX\\\\images\\\\XXXX\\\\crop\\\\460x349\\\\nt\": inserting into folders: executing `INSERT INTO `folders` (`created_at`, `mod_time`, `parent_folder_id`, `path`, `updated_at`, `zip_file_id`) VALUES (?, ?, ?, ?, ?, ?)` [[2022-08-04 10:47:06.7598688 -0400 EDT m=+5598.143112701 2021-05-09 14:02:00 -0400 EDT 43586 I:/Sites/XXXX/images/XXXX/crop/460x349/nt 2022-08-04 10:47:06.7598688 -0400 EDT m=+5598.143112701 <nil>]]: database is locked"

For context on my stash library size, the migration says it migrated 672809 files, db file size ~6GB, generated folder size ~200GB.

7dJx1qP commented 2 years ago

Had an electrical power outtage midscan, but after starting the scan again it seems like it completed fine and I didn't get any database is locked errors.

One thing I've noticed is there's a significant performance regression on the scenes page load time. My stash is about ~100k scenes and on v1.16.1 it takes less than half a second to load a page size of 20. With the files-refactor it takes between 1.5-2 seconds to load a page size of 20. The difference is much more noticable when I try a page size of 1000. On v1.16.1 it takes about 4-5 seconds while on the files-refactor it's taking around 90 seconds.

The homepage, movies, and markers pages are similarly much slower. The Images, Galleries, Performers, Studios, and Tags pages load times seem the same.

skier233 commented 2 years ago

I found a bug with the statistics Scenes duration stat that shows your total scenes durations. It appears to be querying the scenes table and summing the duration but the scenes table no longer has a duration column.

ferengi82 commented 2 years ago

i tried the new experimental version and get the following error at the schema migration

An error occurred migrating the database to the latest schema version. The backup database file was automatically renamed to restore the database. error performing migration: running post migrations for schema version 32: migrating files: UNIQUE constraint failed: files.parent_folder_id, files.basename

djbtwothousandtwenty commented 2 years ago

one of my filters for Galleries has broken. I've made different views, where i've got some paths on my nas systems that contain "favorited models" and some paths contain stuff that's fully unsorted or only sorted by studio. I filter those out with an exclude filter like:

path : excludes : \Studios\

as the path of the files is for example file://nasname/foldername/Studios/studioname/performername/somefile.zip I did try /Studios/ as well, no difference. This worked on 0.16.1. I'm running this on Windows with the shares on the NAS mounted via SMB. The same filter still works for Scenes I just noticed.

For scenes a similar filter I use, again on Path:

path: includes: ".and." combined with a filter on performer count (1 or less) to find some scenes where only 1 of multiple performers has been tagged, so i can then go and find the other performers and manually add them to the scene and improve the metadata. This one also does not work anymore.

djbtwothousandtwenty commented 2 years ago

There's a bug in the Settings on the Tasks tab. On setting any of the settings for Library Scan, and hitting the Scan button, all settings for Library Scan get disabled.

djbtwothousandtwenty commented 2 years ago

Scan settings issue has been fixed for me with #https://github.com/stashapp/stash/pull/2811

ferengi82 commented 2 years ago

with the new version I got this error:

An error occurred migrating the database to the latest schema version. The backup database file was automatically renamed to restore the database. error performing migration: running post migrations for schema version 32: migrating files: migrating file /storage/xxx/000-studio/2000-11-07 - Adriana Sage Jon Dough in Bring'um Young 1 - Adrianna Sage Adriana Sage.m4v: UNIQUE constraint failed: files.parent_folder_id, files.basename

next, I reinstalled the last stable, renamed the file (removed the ' in bring'um), clean and full rescan.

then back to experimental, now the migration worked.

djbtwothousandtwenty commented 2 years ago

I got about a dozen of these errors while running my first full library scan post-migration:

level=error msg="updating files: executing `UPDATE `files` SET `basename`=?,`created_at`=?,`id`=?,`mod_time`=?,`parent_folder_id`=?,`size`=?,`updated_at`=?,`zip_file_id`=? WHERE (`files`.`id` = ?)` [[XXXXXXXXXX.avi 2022-08-04 09:36:28.7004594 -0400 -0400 672814 2022-06-30 09:30:11 -0400 -0400 8976 495341568 2022-08-04 09:36:28.7004594 -0400 -0400 <nil> 672814]]: database is locked"

And then a few of these errors:

level=error msg="error processing \"I:\\\\Sites\\\\XXXX\\\\images\\\\XXXX\\\\crop\\\\460x349\\\\nt\": creating folder \"I:\\\\Sites\\\\XXXX\\\\images\\\\XXXX\\\\crop\\\\460x349\\\\nt\": inserting into folders: executing `INSERT INTO `folders` (`created_at`, `mod_time`, `parent_folder_id`, `path`, `updated_at`, `zip_file_id`) VALUES (?, ?, ?, ?, ?, ?)` [[2022-08-04 10:47:06.7598688 -0400 EDT m=+5598.143112701 2021-05-09 14:02:00 -0400 EDT 43586 I:/Sites/XXXX/images/XXXX/crop/460x349/nt 2022-08-04 10:47:06.7598688 -0400 EDT m=+5598.143112701 <nil>]]: database is locked"

For context on my stash library size, the migration says it migrated 672809 files, db file size ~6GB, generated folder size ~200GB.

i've got a few of these database locked errors as well during the first scan after db migration:

time="2022-08-08 18:34:01" level=error msg="updating files: executingUPDATE files SET basename=?,created_at=?,id=?,mod_time=?,parent_folder_id=?,size=?,updated_at=?,zip_file_id=? WHERE (files.id = ?)[[videoname.4k.mp4 2022-08-08 18:30:41.7705847 +0200 +0200 7095946 2022-08-08 08:19:25 +0200 +0200 91365 1892803057 2022-08-08 18:30:41.7705847 +0200 +0200 <nil> 7095946]]: database is locked"

7dJx1qP commented 2 years ago

Tried the latest file-refactor release and still getting slow performance. Using the network tab to measure times now, on v1.16.1 with 1000 page size and random sort order I get graphql requests taking ~3s. On the files-refactor it's ~1.2min.

I looked through the logs with log level trace and it looks like the total logged SQL query time is neglible compared to the overall time taken and the slow performance is coming from something else.

djbtwothousandtwenty commented 2 years ago

i see the same issue here. on 0.1.16 im able to rescan my full library (17 tb total spread over 2 nas devices on smb in windows) within 8 hours, on the refactor version it's been going for over 2 days and is about 20% in. I don't want to complain about performance, but there's definitely a performance regression somewhere. I've tried to find if there were any slow queries being logged, but most, if not all db calls that end up being traced seem to be around 513.1µs. Im not sure the issue is in the DB side of things, or the query limiting performance is not being traced.

I've experimented with the thread count in settings, going between 0 and increments of 2 up to 20 to see if that makes a difference, my cpu is a 12 core (24 with hyperthreading), but that hardly seems to make a difference. in the 0.16.1 version i've had it set to 9 threads.

An observation is that only during scanning/thumbprinting of large videofiles, does the network adapter ever get fully taxed up to the 1gbit maximum, where with 0.16.1 that'd be happening regardless of whether it was processing video files or images. Processing just galleries consisting of jpg files (no zips) on 0.16.1 would get network speeds between 700mbit and 1000mbit on my environment. So far the initial scan has not reached the gallery zip files yet, so i still need to see if those speed up the process.

djbtwothousandtwenty commented 2 years ago

a Clean, empty db makes a world of difference. Comparing 0.16.0 to filesrefactor as below, with the same dataset (109gb of mixed video both sd, 1080p and 4k and image content without zips consisting of 23.713 files and 378 folders.) on the same hardware and with the exact same configuration:

version | clean/existing | First scan | Second scan | first scan difference | second scan difference -- | -- | -- | -- | -- | -- v0.16.1-1-gcba0fddf | clean | 0:29:07 | 0:00:09 |   |   v0.16.1-1-gcba0fddf | existing | 0:30:32 | 0:00:11 | 0:01:25 | 0:00:02 files-refactor-release-1-g27007365 | clean | 0:59:33 | 0:00:40 | 0:30:26 | 0:00:31 files-refactor-release-1-g27007365 | existing | 1:21:56 | 1:30:10 | 0:52:49 | 1:30:01

Edited: 4 runs over same content in 2 configurations (deleted db and with the existing "production db from 0.16.1).

skier233 commented 2 years ago

I'm experiencing an issue where when I try to delete scenes, the source files are not deleted.

MrX292 commented 2 years ago

some scene sort options do not work like File Size, Interactive and Interactive speed. i get for File Size: error finding IDs: running query: SELECT DISTINCT scenes.id FROM scenes ORDER BY cast(scenes.size as integer) DESC LIMIT 40 OFFSET 0 [[]]: error executing `SELECT DISTINCT scenes.id FROM scenes ORDER BY cast(scenes.size as integer) DESC LIMIT 40 OFFSET 0 ` [[]]: no such column: scenes.size Interactive: error finding IDs: running query: SELECT DISTINCT scenes.id FROM scenes ORDER BY scenes.interactive DESC LIMIT 40 OFFSET 0 [[]]: error executing `SELECT DISTINCT scenes.id FROM scenes ORDER BY scenes.interactive DESC LIMIT 40 OFFSET 0 ` [[]]: no such column: scenes.interactive Interactive speed: error finding IDs: running query: SELECT DISTINCT scenes.id FROM scenes ORDER BY scenes.interactive_speed DESC LIMIT 40 OFFSET 0 [[]]: error executing `SELECT DISTINCT scenes.id FROM scenes ORDER BY scenes.interactive_speed DESC LIMIT 40 OFFSET 0 ` [[]]: no such column: scenes.interactive_speed

djbtwothousandtwenty commented 2 years ago

a Clean, empty db makes a world of difference. Comparing 0.16.0 to filesrefactor as below, with the same dataset (109gb of mixed video both sd, 1080p and 4k and image content without zips consisting of 23.713 files and 378 folders.) on the same hardware and with the exact same configuration:

version clean/existing First scan Second scan first scan difference second scan difference v0.16.1-1-gcba0fddf clean 0:29:07 0:00:09     v0.16.1-1-gcba0fddf existing 0:30:32 0:00:11 0:01:25 0:00:02 files-refactor-release-1-g27007365 clean 0:59:33 0:00:40 0:30:26 0:00:31 files-refactor-release-1-g27007365 existing 1:21:56 1:30:10 0:52:49 1:30:01 Edited: 4 runs over same content in 2 configurations (deleted db and with the existing "production db from 0.16.1).

think i might have solved this issue, ran an ANALYZE and PRAGMA OPTIMIZE command on my sqlite db while it was slowly chugging along, right after that the speed increased exponentially, back to 0.16.1 speeds (at least visually in the UI). Will leave it running and then see what's different in the db compared to the non analyzed / optimized versions.

djbtwothousandtwenty commented 2 years ago

I'm experiencing an issue where when I try to delete scenes, the source files are not deleted.

i can not reproduce this on my environment, deleting scenes works fine for me. It seems to create a whateverthefilenamewas.mp4.delete file which i then find in my recycle bin on the nas.

djbtwothousandtwenty commented 2 years ago

some scene sort options do not work like File Size, Interactive and Interactive speed. i get for File Size: error finding IDs: running query: SELECT DISTINCT scenes.id FROM scenes ORDER BY cast(scenes.size as integer) DESC LIMIT 40 OFFSET 0 [[]]: error executing `SELECT DISTINCT scenes.id FROM scenes ORDER BY cast(scenes.size as integer) DESC LIMIT 40 OFFSET 0 ` [[]]: no such column: scenes.size Interactive: error finding IDs: running query: SELECT DISTINCT scenes.id FROM scenes ORDER BY scenes.interactive DESC LIMIT 40 OFFSET 0 [[]]: error executing `SELECT DISTINCT scenes.id FROM scenes ORDER BY scenes.interactive DESC LIMIT 40 OFFSET 0 ` [[]]: no such column: scenes.interactive Interactive speed: error finding IDs: running query: SELECT DISTINCT scenes.id FROM scenes ORDER BY scenes.interactive_speed DESC LIMIT 40 OFFSET 0 [[]]: error executing `SELECT DISTINCT scenes.id FROM scenes ORDER BY scenes.interactive_speed DESC LIMIT 40 OFFSET 0 ` [[]]: no such column: scenes.interactive_speed

confirmed on my environment

djbtwothousandtwenty commented 2 years ago

̶i̶̶̶'̶̶̶m̶̶̶ ̶̶̶m̶̶̶i̶̶̶s̶̶̶s̶̶̶i̶̶̶n̶̶̶g̶̶̶ ̶̶̶t̶̶̶h̶̶̶e̶̶̶ ̶̶̶"̶̶̶D̶̶̶u̶̶̶p̶̶̶l̶̶̶i̶̶̶c̶̶̶a̶̶̶t̶̶̶e̶̶̶d̶̶̶ ̶̶̶(̶̶̶p̶̶̶h̶̶̶a̶̶̶s̶̶̶h̶̶̶)̶̶̶"̶̶̶ ̶̶̶f̶̶̶i̶̶̶l̶̶̶t̶̶̶e̶̶̶r̶̶̶ ̶̶̶o̶̶̶p̶̶̶t̶̶̶i̶̶̶o̶̶̶n̶̶̶ ̶̶̶o̶̶̶n̶̶̶ ̶̶̶I̶̶̶m̶̶̶a̶̶̶g̶̶̶e̶̶̶s̶̶̶.̶̶̶ ̶̶̶C̶̶̶o̶̶̶u̶̶̶l̶̶̶d̶̶̶ ̶̶̶b̶̶̶e̶̶̶ ̶̶̶s̶̶̶i̶̶̶n̶̶̶c̶̶̶e̶̶̶ ̶̶̶a̶̶̶l̶̶̶l̶̶̶ ̶̶̶i̶̶̶m̶̶̶a̶̶̶g̶̶̶e̶̶̶s̶̶̶ ̶̶̶d̶̶̶o̶̶̶ ̶̶̶a̶̶̶p̶̶̶p̶̶̶e̶̶̶a̶̶̶r̶̶̶ ̶̶̶t̶̶̶o̶̶̶ ̶̶̶h̶̶̶a̶̶̶v̶̶̶e̶̶̶ ̶̶̶a̶̶̶ ̶̶̶f̶̶̶i̶̶̶n̶̶̶g̶̶̶e̶̶̶r̶̶̶p̶̶̶r̶̶̶i̶̶̶n̶̶̶t̶̶̶ ̶̶̶(̶̶̶M̶̶̶d̶̶̶5̶̶̶ ̶̶̶b̶̶̶a̶̶̶s̶̶̶e̶̶̶d̶̶̶)̶̶̶ ̶̶̶w̶̶̶h̶̶̶i̶̶̶c̶̶̶h̶̶̶ ̶̶̶i̶̶̶s̶̶̶ ̶̶̶n̶̶̶o̶̶̶t̶̶̶ ̶̶̶p̶̶̶h̶̶̶a̶̶̶s̶̶̶h̶̶̶.̶̶̶ ̶̶̶N̶̶̶o̶̶̶t̶̶̶ ̶̶̶s̶̶̶u̶̶̶r̶̶̶e̶̶̶ ̶̶̶i̶̶̶f̶̶̶ ̶̶̶p̶̶̶h̶̶̶a̶̶̶s̶̶̶h̶̶̶ ̶̶̶i̶̶̶s̶̶̶ ̶̶̶i̶̶̶n̶̶̶ ̶̶̶t̶̶̶h̶̶̶e̶̶̶ ̶̶̶c̶̶̶a̶̶̶r̶̶̶d̶̶̶s̶̶̶ ̶̶̶f̶̶̶o̶̶̶r̶̶̶ ̶̶̶i̶̶̶m̶̶̶a̶̶̶g̶̶̶e̶̶̶s̶̶̶ ̶̶̶a̶̶̶n̶̶̶d̶̶̶ ̶̶̶g̶̶̶a̶̶̶l̶̶̶l̶̶̶e̶̶̶r̶̶̶i̶̶̶e̶̶̶s̶̶̶.̶̶̶ ̶̶̶W̶̶̶i̶̶̶t̶̶̶h̶̶̶ ̶̶̶g̶̶̶a̶̶̶l̶̶̶l̶̶̶e̶̶̶r̶̶̶i̶̶̶e̶̶̶s̶̶̶ ̶̶̶i̶̶̶t̶̶̶ ̶̶̶c̶̶̶o̶̶̶u̶̶̶l̶̶̶d̶̶̶ ̶̶̶b̶̶̶e̶̶̶ ̶̶̶i̶̶̶n̶̶̶t̶̶̶e̶̶̶r̶̶̶e̶̶̶s̶̶̶t̶̶̶i̶̶̶n̶̶̶g̶̶̶ ̶̶̶t̶̶̶o̶̶̶ ̶̶̶f̶̶̶i̶̶̶r̶̶̶s̶̶̶t̶̶̶ ̶̶̶g̶̶̶e̶̶̶n̶̶̶e̶̶̶r̶̶̶a̶̶̶t̶̶̶e̶̶̶ ̶̶̶a̶̶̶ ̶̶̶o̶̶̶n̶̶̶e̶̶̶ ̶̶̶i̶̶̶m̶̶̶a̶̶̶g̶̶̶e̶̶̶ ̶̶̶m̶̶̶o̶̶̶s̶̶̶a̶̶̶i̶̶̶c̶̶̶ ̶̶̶o̶̶̶f̶̶̶ ̶̶̶a̶̶̶l̶̶̶l̶̶̶ ̶̶̶i̶̶̶m̶̶̶a̶̶̶g̶̶̶e̶̶̶s̶̶̶ ̶̶̶i̶̶̶n̶̶̶ ̶̶̶t̶̶̶h̶̶̶e̶̶̶ ̶̶̶g̶̶̶a̶̶̶l̶̶̶l̶̶̶e̶̶̶r̶̶̶y̶̶̶ ̶̶̶s̶̶̶o̶̶̶ ̶̶̶y̶̶̶o̶̶̶u̶̶̶ ̶̶̶e̶̶̶n̶̶̶d̶̶̶ ̶̶̶u̶̶̶p̶̶̶ ̶̶̶w̶̶̶i̶̶̶t̶̶̶h̶̶̶ ̶̶̶a̶̶̶ ̶̶̶s̶̶̶i̶̶̶m̶̶̶i̶̶̶l̶̶̶a̶̶̶r̶̶̶ ̶̶̶o̶̶̶b̶̶̶j̶̶̶e̶̶̶c̶̶̶t̶̶̶ ̶̶̶a̶̶̶s̶̶̶ ̶̶̶w̶̶̶i̶̶̶t̶̶̶h̶̶̶ ̶̶̶t̶̶̶h̶̶̶e̶̶̶ ̶̶̶s̶̶̶c̶̶̶e̶̶̶n̶̶̶e̶̶̶s̶̶̶,̶̶̶ ̶̶̶a̶̶̶n̶̶̶d̶̶̶ ̶̶̶t̶̶̶h̶̶̶e̶̶̶n̶̶̶ ̶̶̶r̶̶̶u̶̶̶n̶̶̶ ̶̶̶t̶̶̶h̶̶̶e̶̶̶ ̶̶̶p̶̶̶h̶̶̶a̶̶̶s̶̶̶h̶̶̶ ̶̶̶l̶̶̶o̶̶̶g̶̶̶i̶̶̶c̶̶̶ ̶̶̶o̶̶̶v̶̶̶e̶̶̶r̶̶̶ ̶̶̶i̶̶̶t̶̶̶ ̶̶̶t̶̶̶o̶̶̶ ̶̶̶i̶̶̶d̶̶̶e̶̶̶n̶̶̶t̶̶̶i̶̶̶f̶̶̶y̶̶̶ ̶̶̶s̶̶̶i̶̶̶m̶̶̶i̶̶̶l̶̶̶a̶̶̶r̶̶̶/̶̶̶v̶̶̶e̶̶̶r̶̶̶y̶̶̶ ̶̶̶s̶̶̶i̶̶̶m̶̶̶i̶̶̶l̶̶̶a̶̶̶r̶̶̶/̶̶̶e̶̶̶x̶̶̶a̶̶̶c̶̶̶t̶̶̶l̶̶̶y̶̶̶ ̶̶̶t̶̶̶h̶̶̶e̶̶̶ ̶̶̶s̶̶̶a̶̶̶m̶̶̶e̶̶̶ ̶̶̶g̶̶̶a̶̶̶l̶̶̶l̶̶̶e̶̶̶r̶̶̶i̶̶̶e̶̶̶s̶̶̶ ̶̶̶b̶̶̶a̶̶̶s̶̶̶e̶̶̶d̶̶̶ ̶̶̶o̶̶̶n̶̶̶ ̶̶̶c̶̶̶o̶̶̶n̶̶̶t̶̶̶e̶̶̶n̶̶̶t̶̶̶.̶̶̶ ̶̶̶g̶̶̶u̶̶̶e̶̶̶s̶̶̶s̶̶̶ ̶̶̶t̶̶̶h̶̶̶a̶̶̶t̶̶̶ ̶̶̶w̶̶̶o̶̶̶u̶̶̶l̶̶̶d̶̶̶ ̶̶̶o̶̶̶n̶̶̶l̶̶̶y̶̶̶ ̶̶̶w̶̶̶o̶̶̶r̶̶̶k̶̶̶ ̶̶̶o̶̶̶n̶̶̶ ̶̶̶g̶̶̶a̶̶̶l̶̶̶l̶̶̶e̶̶̶r̶̶̶i̶̶̶e̶̶̶s̶̶̶ ̶̶̶t̶̶̶h̶̶̶a̶̶̶t̶̶̶ ̶̶̶h̶̶̶a̶̶̶v̶̶̶e̶̶̶ ̶̶̶t̶̶̶h̶̶̶e̶̶̶ ̶̶̶s̶̶̶a̶̶̶m̶̶̶e̶̶̶ ̶̶̶i̶̶̶m̶̶̶a̶̶̶g̶̶̶e̶̶̶ ̶̶̶c̶̶̶o̶̶̶u̶̶̶n̶̶̶t̶̶̶ ̶̶̶a̶̶̶n̶̶̶d̶̶̶ ̶̶̶o̶̶̶r̶̶̶d̶̶̶e̶̶̶r̶̶̶,̶̶̶ ̶̶̶b̶̶̶u̶̶̶t̶̶̶ ̶̶̶i̶̶̶t̶̶̶'̶̶̶d̶̶̶ ̶̶̶b̶̶̶e̶̶̶ ̶̶̶a̶̶̶ ̶̶̶g̶̶̶o̶̶̶o̶̶̶d̶̶̶ ̶̶̶s̶̶̶t̶̶̶a̶̶̶r̶̶̶t̶̶̶?̶̶̶ ̶̶̶ ̶̶̶ ̶a̶̶̶n̶̶̶y̶̶̶w̶̶̶a̶̶̶y̶̶̶,̶̶̶ ̶̶̶w̶̶̶h̶̶̶a̶̶̶t̶̶̶ ̶̶̶i̶̶̶ ̶̶̶g̶̶̶u̶̶̶e̶̶̶s̶̶̶s̶̶̶ ̶̶̶i̶̶̶'̶̶̶m̶̶̶ ̶̶̶s̶̶̶a̶̶̶y̶̶̶i̶̶̶n̶̶̶g̶̶̶ ̶̶̶i̶̶̶s̶̶̶:̶̶̶ ̶̶̶i̶̶̶'̶̶̶m̶̶̶ ̶̶̶m̶̶̶i̶̶̶s̶̶̶s̶̶̶i̶̶̶n̶̶̶g̶̶̶ ̶̶̶a̶̶̶ ̶̶̶f̶̶̶i̶̶̶l̶̶̶t̶̶̶e̶̶̶r̶̶̶ ̶̶̶t̶̶̶o̶̶̶ ̶̶̶f̶̶̶i̶̶̶g̶̶̶u̶̶̶r̶̶̶e̶̶̶ ̶̶̶o̶̶̶u̶̶̶t̶̶̶ ̶̶̶w̶̶̶h̶̶̶i̶̶̶c̶̶̶h̶̶̶ ̶̶̶d̶̶̶u̶̶̶p̶̶̶l̶̶̶i̶̶̶c̶̶̶a̶̶̶t̶̶̶e̶̶̶ ̶̶̶i̶̶̶m̶̶̶a̶̶̶g̶̶̶e̶̶̶s̶̶̶ ̶̶̶i̶̶̶ ̶̶̶h̶̶̶a̶̶̶v̶̶̶e̶̶̶.̶̶̶

edit: found it, you can do this with filter "file count" larger than 1

skier233 commented 2 years ago

I'm experiencing an issue where when I try to delete scenes, the source files are not deleted.

i can not reproduce this on my environment, deleting scenes works fine for me. It seems to create a whateverthefilenamewas.mp4.delete file which i then find in my recycle bin on the nas.

Hmm. On my system the file I was trying to delete is on a different drive than where stash is running from. Maybe this could be a factor?

7dJx1qP commented 2 years ago

I'm experiencing an issue where when I try to delete scenes, the source files are not deleted.

i can not reproduce this on my environment, deleting scenes works fine for me. It seems to create a whateverthefilenamewas.mp4.delete file which i then find in my recycle bin on the nas.

Hmm. On my system the file I was trying to delete is on a different drive than where stash is running from. Maybe this could be a factor?

Is the "Delete file (and funscript)" option selected when you delete?

skier233 commented 2 years ago

Yea. And I tried it a few times and it seems to always have the same behavior.

djbtwothousandtwenty commented 2 years ago

small bug, the folder crawler finds folders which have an image that is within the excluded filenames list, but still adds a folder for it:

Performerfoldername doesn't exist. Creating new folder entry...

it should probably check whether the content that triggered the folder crawler to find the folder, was only a file that's not wanted in the db

djbtwothousandtwenty commented 2 years ago

on re-scanning zip galleries i mainly get these:

error processing "\\nas\galleryfolder\galleryfilename.zip": adding file to gallery: inserting into galleries_files: executing INSERT INTOgalleries_files(gallery_id,primary,file_id) VALUES (114849, 0, 7072240) [[]]: UNIQUE constraint failed: galleries_files.gallery_id, galleries_files.file_id

djbtwothousandtwenty commented 2 years ago

on the performer page > Galleries view, sorting by Path (ascending or descending) does not change the sort order anymore. This is also the case on the Galleries page (/galleries?sortby=path&sortdir=desc)

djbtwothousandtwenty commented 2 years ago

Using the clean task, on deleted scenes (deleted outside of stash)

2022-08-16 23:01:09 Error
Error deleting folder "\\nas\folder\filename.1080p.MP4" from database: destroying folders: executing DELETE FROMfoldersWHERE (folders.idIN (92216)) [[]]: FOREIGN KEY constraint failed

this also occurs on zip files that no longer exist.

djbtwothousandtwenty commented 2 years ago

Scan settings issue has been fixed for me with ##2811

i've found one more location where this still happens: if I set the generate options for Scan task all to ON, but do not hit the scan buttons, it seems to save the setting correctly. it resets those specific sliders to the OFF position immediately after hitting the Generate button though

djbtwothousandtwenty commented 2 years ago

the autotagger seems to increasingly take longer time over each set of 1000 items it processes:

SLOW SQL [302.7667ms]: SELECT DISTINCT images.id FROM images LEFT JOIN images_files ON images_files.image_id = images.id LEFT JOIN files ON images_files.file_id = files.id LEFT JOIN folders ON files.parent_folder_id = folders.id WHERE (((folders.path LIKE ? AND files.basename LIKE ?) OR (folders.path LIKE ?)) AND (images.organized = 0)) LIMIT 1000 OFFSET 594000 , args: [//nas/foldername % //nas/foldername/%]

^ that's after about 600000 images

djbtwothousandtwenty commented 2 years ago

Galleries no longer get tagged for me, no errors in the logs. Images inside the galleries do get tagged correctly, but not the folder based gallery. Manual scrape with Autotag does give the correct tags, so there's a manual workaround :)

ferengi82 commented 2 years ago

the renamerOnUpdate doesn't work with this release, maybe there are other plugins that are affected. https://github.com/stashapp/CommunityScripts/issues/73

ferengi82 commented 2 years ago

when you try to submit a new performer to stash-box I got an error

Submission failed
already in transaction
scruffynerf commented 2 years ago

the renamerOnUpdate doesn't work with this release, maybe there are other plugins that are affected. stashapp/CommunityScripts#73

Wouldn't be surprised, it does direct SQL. This is why using GraphQL is safer. Yes, renamerOnUpdate and other scrapers/plugins/scripts that use sqlite to talk directly will likely all break, and need to be rewritten. That's expected, with a major rewrite like this.

WithoutPants commented 2 years ago

Thanks for the continued feedback.

2849 fixes the following reported issues:

the autotagger seems to increasingly take longer time over each set of 1000 items it processes:

This is an issue with the existing system as well. It requires some performance tuning that I won't be addressing in the short term.

on re-scanning zip galleries i mainly get these:

error processing "\nas\galleryfolder\galleryfilename.zip": adding file to gallery: inserting into galleries_files: executing INSERT INTOgalleries_files(gallery_id,primary,file_id) VALUES (114849, 0, 7072240) [[]]: UNIQUE constraint failed: galleries_files.gallery_id, galleries_files.file_id

Using the clean task, on deleted scenes (deleted outside of stash)

2022-08-16 23:01:09 Error Error deleting folder "\nas\folder\filename.1080p.MP4" from database: destroying folders: executing DELETE FROMfoldersWHERE (folders.idIN (92216)) [[]]: FOREIGN KEY constraint failed

this also occurs on zip files that no longer exist.

I haven't been able to reproduce these issues. I'll probably need a more specific scenario.

I also haven't been able to reproduce the file deletion issue.

MrX292 commented 2 years ago

i cant use part of the scene path in Scene Filename Parser in the Filename Pattern like X:\\Downloads\\p_v\\Studio\\{title}.{ext} for Windows anymore. Which worked in v0.16.1-1-gcba0fddf

djbtwothousandtwenty commented 2 years ago

unfortunately, the issue with the cleaner not cleaning, remains. I can post a log file, but the main way i can reproduce this is by:

what i see in the logs is:

level=info msg="Folder not found. Marking to clean: "\nasname\foldername\models\performername\imagesetname.XXX.IMAGESET""

followed after a few minutes by:

level=error msg="Error deleting folder "\\nasname\foldername\models\performername\imagesetname.XXX.IMAGESET" from database: destroying folders: executing DELETE FROM folders WHERE (folders.id IN (37762)) [[]]: FOREIGN KEY constraint failed"

there's a difference between the folder it mentions the first time while correctly noticing the folder is missing, and the second time while trying to remove it, but im not sure if that's just the way the logger prints the path, or if it's actually trying to delete that quoted path literally, because that's having a few too many \ characters all of this on windows, with a smb share all on my "production" db as well, that i copied over from 0.16.1, will see if i can reproduce the same issue with a fully clean new db as well

going to a scene in stash, and deleting it manually from stash, seems to result in a deletion, as in: the scene is no longer visible in the Scenes overview page as expected. However the cleaner still seems to know about it, and still tries to remove it, so i guess there's still a folder or path reference in the db for it for gallery, this is exactly the same, going to look for imagesetname.XXX.IMAGESET yields no results in the galleries view also no results for that gallery name in the images view so it appears visually in stash, that there's no problem, except the cleaner still has a reference to it that it's not able to remove.

WithoutPants commented 2 years ago

The latest build changes the way that paths are stored in the folders table for Windows users. Windows users will need to re-migrate from the version 31 schema, or manually adjust the path column to user \ instead of / as path separators.

This should address the filename parser regression and should fix autotagging on Windows systems. I'm hoping that this will also fix the cleaning issue reported, but please retest and let me know.

WithoutPants commented 2 years ago

The latest build should properly fix the clean issue reported above. It also re-introduces the import/export functionality.

djbtwothousandtwenty commented 2 years ago

https://github.com/stashapp/stash/issues/2737#issuecomment-1222139966 is fixed for me now as well.

I did find 2 more things, both either directly or somewhat related to the cleaner:

  1. if i remove all library paths, and not touch the content, the cleaner will not clean anything. if all library paths are gone, i'd think the cleaner should end up with an empty database where it concerns scenes, galleries and images as it has paths for those that no longer "should be there".
  2. I've found another bug related to the library paths. I can delete all paths including the last path, and the ui shows it empty. if i refresh (f5) or navigate away and back to settings, the last deleted library path is back again like it was before deleting. There's no trace or other log entries related to this to be found in the logs.
djbtwothousandtwenty commented 2 years ago

with the latest build files-refactor-release-1-g0aad77d7

have got 2 very similar (but not 100% identical) scene files, that i found with the scene duplicates finder. one file is higher resolution and comes in at 19mb, the other has a different name and is smaller, so i want to delete it. Hitting "delete" and selecting that i also want to delete the file itself + all generated stuff, i get an error. log only spits out:

time="2022-09-01 12:31:14" level=error msg="relationship has not been loaded"

got another new error in the log, unfortunately it does not say which tag or which image file it had issues with:

time="2022-09-01 13:33:17" level=info msg="Autotagging images..." time="2022-09-01 13:33:19" level=error msg="error tagging image performers for : error executing SELECT performers.* FROM performers WHERE ignore_auto_tag = 0 AND () [[]]: near ")": syntax error"

with tracelogging enabled the trace just before the error is:

time="2022-09-01 15:01:40" level=trace msg="SQL [0s]: SELECT performers.* FROM performers WHERE ignore_auto_tag = 0 AND (), args: []" time="2022-09-01 15:01:40" level=error msg="error tagging image performers for : error executing SELECT performers.* FROM performers WHERE ignore_auto_tag = 0 AND () [[]]: near ")": syntax error"

last one is a small visual thing: when starting stash, the settings of library scan, do not stay saved, they keep getting reset to all off.

performance of queries in views is much improved, scanning for new/changed files is also much faster, very nice!

skier233 commented 2 years ago

Trying to delete scenes that are currently open (even if paused) in the scene viewer yields the following error: Capture

Instead, any processes holding the file should be terminated on deletion.

DogmaDragon commented 2 years ago

Trying to delete scenes that are currently open (even if paused) in the scene viewer yields the following error: Capture

Instead, any processes holding the file should be terminated on deletion.

Tested and it works in v0.16.1-8e222ae3.

WithoutPants commented 2 years ago

Trying to delete scenes that are currently open (even if paused) in the scene viewer yields the following error: Instead, any processes holding the file should be terminated on deletion.

I'm not able to reproduce this. Is this doing a direct stream or live transcoding?

skier233 commented 2 years ago

Trying to delete scenes that are currently open (even if paused) in the scene viewer yields the following error: Instead, any processes holding the file should be terminated on deletion.

I'm not able to reproduce this. Is this doing a direct stream or live transcoding?

@WithoutPants It's on direct stream with mp4 files (at least). Maybe something to try would be having multiple stash tabs open too? I usually have one stash tab open with a list of scenes and individual stash tabs for each scene.

WithoutPants commented 2 years ago

@WithoutPants It's on direct stream with mp4 files (at least). Maybe something to try would be having multiple stash tabs open too? I usually have one stash tab open with a list of scenes and individual stash tabs for each scene.

Still unable to reproduce. What's the exact steps to reproduce? Is E:\ in the above screenshot on the local machine or over network?

skier233 commented 2 years ago

@WithoutPants It's on direct stream with mp4 files (at least). Maybe something to try would be having multiple stash tabs open too? I usually have one stash tab open with a list of scenes and individual stash tabs for each scene.

Still unable to reproduce. What's the exact steps to reproduce? Is E:\ in the above screenshot on the local machine or over network?

E: is on the local machine. The steps is basically just opening a tab for a scene and starting to play the scene and then delete. Make sure you are trying a full delete with both options selected. Also make sure the video plays for like 30 seconds or so before you try it. Im using brave browser though I doubt that matters.

Edit: Also, I have the scrubber sprites at the bottom generated to be able to navigate through the video. Perhaps the scrubber sprites are locking those from being deleted?

WithoutPants commented 2 years ago

This is the procedure I've used to try to reproduce this:

I assume you can remove scene files successfully if you do so without playing the scene (in the scene page and the scenes page)?

skier233 commented 2 years ago

This is the procedure I've used to try to reproduce this:

  • scan file and generate sprites and preview
  • locate file in scenes page, open scene in new tab
  • play scene for at least 30 seconds - I've also tried scrubbing around the scene randomly
  • on the scene page, click on the Edit tab, then the Delete button
  • tick both Delete File and Delete generated options and click the Delete button
  • system indicates scene was deleted successfully and returns to scenes page
  • verified file was removed from the filesystem

I assume you can remove scene files successfully if you do so without playing the scene (in the scene page and the scenes page)?

Yea I can delete them no problem if I delete before playing the video.

djbtwothousandtwenty commented 2 years ago

i've also tried to reproduce the issue @skier233 reported, but so far i've not been able to either. This is with items scanned (and fully phash / preview generated) beforehand, deleting while playing stops the file and deletes it succesfully and returns me back to the scenes view. in my case using chrome, on windows 11, with the files mounted on a smb share on a nas. I can see that the delete action first renames the file to add ".delete" after the extension and then proceeds to delete it.

djbtwothousandtwenty commented 2 years ago

during the scan after re-migration for version files-refactor-release-1-gcfc8222b, it notices as gallery folder as not yet existing (it's one that got a bit of the path changed). it creates an empty gallery, with no files in it, even though the scan log indicates that it found all the files as pre-existing files based on thumbprint, knows both old path and new path and says it's updating the path for the image.

log entries:

galleryname.XXX.IMAGESET doesn't exist. Creating new folder entry... Calculating fingerprints for galleryname.XXX.IMAGESET\001.jpg ..." \\nas\folder\performers\oldperformername\galleryname.XXX.IMAGESET\001.jpg moved to \\nas\folder\performers\newperformername\galleryname.XXX.IMAGESET\001.jpg. Updating path..."

it does this calculating, "moved to" for all 119 files correctly

looking for that gallery through the galleries view, i've got the "old one" and the new empty one, with the old gallery info pointing to the old path: \\nas\folder\performers\oldperformername\galleryname.XXX.IMAGESET\, but it has all 119 images and the images are all loading correctly. The new empty gallery is pointing to the new location: \\nas\folder\performers\newperformername\galleryname.XXX.IMAGESET\

the individual images are only connected to the old gallery, and they have been correctly updated to reflect the new path. so i think there might be a bug with the gallery update statement, where it creates a new one, instead of updates the existing one to the new path like it does for the images. this specific gallery had no duplicates either as far as i'm aware

After re-migration, initial scan, clean and another scan, i've now got 45 empty galleries like this

skier233 commented 2 years ago

i've also tried to reproduce the issue @skier233 reported, but so far i've not been able to either. This is with items scanned (and fully phash / preview generated) beforehand, deleting while playing stops the file and deletes it succesfully and returns me back to the scenes view. in my case using chrome, on windows 11, with the files mounted on a smb share on a nas. I can see that the delete action first renames the file to add ".delete" after the extension and then proceeds to delete it.

I tested with chrome and verified the same behavior. Also, verified the behavior while on the latest version of the files-refactor branch. Not sure why you'll aren't able to repro when I see the error every time :/

I confirmed that the error still occurs if you only have the checkbox for "delete file and funscript" and not also the generated files checkbox.

skier233 commented 2 years ago

@WithoutPants Aha! I've found what's different and why you aren't able to repro. It only occurs if stash is using an ssl key and running with https. When I disabled https then the issue went away!