search

PaulLereverend / NextcloudDuplicateFinder

Save some space by finding your duplicate files
GNU Affero General Public License v3.0
79 stars 16 forks source link

Endless loading #1

Open pilsnerbeer opened 4 years ago

pilsnerbeer commented 4 years ago

Hi, Im a big fan of the project! However, I am unable to get the app to generate any list of duplicates. I ran duplicates:find-all and was able to find duplicates for all users successfully but the UI doesn't seem to be working. on NC 19 snap , mysql & apache

Picture: endless loading

Screenshot_1

water-man commented 4 years ago

Patience helps! I have some 130Gbyte of photo s and I had (almost) the same experience. Clicked the icon twice before but cancelled after some 5 minutes. During the 3rd attempt I got a (long) phonecall and when I returned: guess what! 1.500 double entries. Nicely outlined with thumbprints and folder spec's . I love the option to remove files right from the result view.

SUGGESTION: (for the GUI), when clicked, offer the directory tree and the possibility to choose from where to start the check. When chosen, have some counter running that displays the amount of files checked and a second counter indicating the number of doubles found. At least we can then see that the app is working and we are not waiting for an endless loop.

(Nextcloud 19 snap on Ubuntu server 20.04 LTS, running on a RaspberryPi 4B 8Gbyte with 2 TB RAID-1.)

pilsnerbeer commented 4 years ago

Thanks for reply. I wonder if the app needs to be kept open all the time while searching? I only have ~ 8 gigs of data on my NC, shouldnt take that long. Will try again.

PaulLereverend commented 4 years ago

Hi, Im a big fan of the project! However, I am unable to get the app to generate any list of duplicates. I ran duplicates:find-all and was able to find duplicates for all users successfully but the UI doesn't seem to be working. on NC 19 snap , mysql & apache

Picture: endless loading

Thanks for reply. I wonder if the app needs to be kept open all the time while searching? I only have ~ 8 gigs of data on my NC, shouldnt take that long. Will try again.

Hi The GUI should take the same time as the command. But indeed it needs to be kept open. Please try to wait a little bit longer, especially if you have a lot of files and small hardware.

PaulLereverend commented 4 years ago

SUGGESTION: (for the GUI), when clicked, offer the directory tree and the possibility to choose from where to start the check.

Hi The command already allows you to specify the desired path. But indeed the interface can be improved.

SUGGESTION: (for the GUI), When chosen, have some counter running that displays the amount of files checked and a second counter indicating the number of doubles found. At least we can then see that the app is working and we are not waiting for an endless loop.

Good idea, I will look into this.

pilsnerbeer commented 4 years ago

@PaulLereverend I let it run for 30 minutes now (kept open) but still nothing. The console took about 5 min max I think.

PaulLereverend commented 4 years ago

@PaulLereverend I let it run for 30 minutes now (kept open) but still nothing. The console took about 5 min max I think.

I will run some tests on NC snap and come back to you asap.

diode-ee commented 4 years ago

I'm experiencing the same issue. When I try running through the GUI it fails after a few seconds and if left overnight, does not complete any task. Log output is:

[PHP] Error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) at /usr/local/www/nextcloud/lib/private/Files/Cache/Cache.php#175

GET /apps/duplicatefinder/files

I'm running this in a TrueNAS jail with 32gb of memory. This seems to happen on a particular login which has over 1TB worth of storage. If I run it on a fresh user, it completed with out errors.

jth134 commented 4 years ago

I am getting the endless loading and looking at the :inspect" I console shows this

$ is deprecated: The global jQuery is deprecated. It will be updated to v2.4 in Nextcloud 20 and v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own.

aleksandrmetik commented 4 years ago

I'm not sure if it's possible to use it when I have 3TB of data and over a billion files. Here you definitely need optimization, background analysis, and output of results, as it is implemented in Face Recognition or Search app.

Bergum commented 4 years ago

When i do a duplicate:find-all i get an error message:

In DAV.php line 863:

Sabre\HTTP\ClientHttpException: Service Unavailable

Pato05 commented 3 years ago

Another suggestion would be to put a cache, so that if the duplicates have already been found (and they have not been deleted), the app would directly show you them, and along with this, you might show the duplicates as they have been found.

volt-systems commented 3 years ago

Hi There, i too got the endless spinner. The problem seems to be different though:

Undefined variable: inputPath at /var/www/nextcloud/apps/duplicatefinder/lib/Command/FindDuplicates.php#69

Here is the full debug backtrace:

Error: Undefined variable: inputPath at /var/www/nextcloud/apps/duplicatefinder/lib/Command/FindDuplicates.php#69 0 /var/www/nextcloud/apps/duplicatefinder/lib/Command/FindDuplicates.php - line 69:OC\Log\ErrorHandler::onError(8, "Undefined variable: inputPath", "/var/www/ne ... p", 69, { user: "*** ... "})

1 /var/www/nextcloud/lib/private/User/Manager.php - line 563:OCA\DuplicateFinder\Command\FindDuplicates->OCA\DuplicateFinder\Command{closure}("** sensiti ... ")

2 /var/www/nextcloud/apps/duplicatefinder/lib/Command/FindDuplicates.php - line 71:OC\User\Manager->callForSeenUsers(Closure {})

3 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php - line 255:OCA\DuplicateFinder\Command\FindDuplicates->execute(Symfony\Comp ... {}, Symfony\Comp ... {})

4 /var/www/nextcloud/core/Command/Base.php - line 169:Symfony\Component\Console\Command\Command->run(Symfony\Comp ... {}, Symfony\Comp ... {})

5 /var/www/nextcloud/3rdparty/symfony/console/Application.php - line 1000:OC\Core\Command\Base->run(Symfony\Comp ... {}, Symfony\Comp ... {})

6 /var/www/nextcloud/3rdparty/symfony/console/Application.php - line 271:Symfony\Component\Console\Application->doRunCommand(OCA\Duplicat ... {}, Symfony\Comp ... {}, Symfony\Comp ... {})

7 /var/www/nextcloud/3rdparty/symfony/console/Application.php - line 147:Symfony\Component\Console\Application->doRun(Symfony\Comp ... {}, Symfony\Comp ... {})

8 /var/www/nextcloud/lib/private/Console/Application.php - line 215:Symfony\Component\Console\Application->run(Symfony\Comp ... {}, Symfony\Comp ... {})

9 /var/www/nextcloud/console.php - line 100:OC\Console\Application->run()

10 /var/www/nextcloud/occ - line 11:require_once("/var/www/ne ... p")

Command Line runs successful, either for all users or only for the one user that also gets this error using the GUI... (-u USER)

Any suggestions?

I'm using NC(P) in version 20.0.2 ... (didn't work with 19.0.5 either)

Update: Analysing the backtrace, I was wondering why the tool was in line 69 at all: It is the "no user branch" if I get that correctly... But I am accessing the tool via the WebUI of an user...?

When I check with Chrome DevTools @Network, I can see that the first call of "duplicatefinder/" is red, no response data. Followed by a second call to "duplicatefinder/" which presents a website "This action requires you to confirm your password"... (See screenshot) duplicatefinder

This info/form however is never shown to me in the GUI - so there is no way to confirm my password - maybe the reason why the tool does not get a user/inputPath and exits with the error above?

Thank you!

fmjensen commented 3 years ago

I experience precisely the same problems and symptoms as @volt-systems describe 😞 My NC version is currently 19.0.6 and it passes all checks on the /index.php/settings/admin/overview page. I hopy you can find time and energy to look into a fix 🙏

🎅 Happy Holidays 🎄

DerekL0576 commented 3 years ago

I get the "Endless loading" with a regular user account in the GUI. Admin accounts and cli work fine. It's a bit frustrating that the menu entry is there for all users when they can't use the app. Using NC21 with version 0.0.5.

tstackhouse commented 3 years ago

I get this when I open the GUI, since it seems like it immediately tried to run the duplicate finder, rather than presenting a dashboard that would let the user manage that. It also seems like it's trying to do the search in the foreground rather than being a background task that can cache the duplicates it found:

image

I'm on NC 20.0.7 and DF 0.0.6 with multiple TB of data in my instance

chrros95 commented 3 years ago

The same problem occurred in my installation as well. As I have a many files this behaviour is quite reasonable. I spent some time finding a different solution and prepared an approach in my fork

The following changes have been implemented:

Next week I would like to spend some time at the controller and the UI to use these changes not only at the cmd.

@PaulLereverend If I then prepare a pull request, are you generally willing to accept this approach?

Mazvy commented 3 years ago

I'm experience something similar I guess. Via the web UI it just shows an endless spinner, in console I get this:

$ sudo -u www-data php occ duplicates:find-all -u USER
Start searching files for USER in path /USER/files
Finished searching files
Duplicates for user "USER" are:

It finished "searching files" in a ~few minutes, but it's stuck on Duplicates for user "USER" are: and not printing anything else for at least 30 minutes now.

iotop doesn't show this process using the disk at all. It's using ~90% of a CPU core.

Edit:

I got it working. As a note for the devs - removing the plugin does not remove oc_dupliactefinder_% tables as well as various configs and other stuff from the database. Removing/reinstalling should, at least in my opinion, result in a fresh install but, well, it doesn't.

Here's the steps I took to get the plugin to start working again:

  1. Disabled the plugin
  2. Removed the plugin
  3. sudo -u www-data php occ maintenance:repair
  4. There were still multiple database tables and various scattered entries containing "duplicatefinder". After exporting the database for backup, I removed every table and entry containing the string %duplicatefinder%
  5. Installed the plugin
  6. sudo -u www-data php occ duplicates:find-all now works as well as the infinite spinner is gone.
chrros95 commented 3 years ago

Thanks for the report @Mazvy. It sounds like you are using the latest version, right? Currently, there is an issue with the pagination, which applies only under certain conditions. It is addressed in PR #58. So if the issue occurs again, please try to apply the changes.

Mazvy commented 3 years ago

Thanks for the report @Mazvy. It sounds like you are using the latest version, right? Currently, there is an issue with the pagination, which applies only under certain conditions. It is addressed in PR #58. So if the issue occurs again, please try to apply the changes.

Yes, latest version (0.0.12).

IIRC during a Nextcloud upgrade Duplicate Finder wasn't available for that version and was disabled automatically. Today I re-enabled and updated, but it didn't work as I mentioned in the previous comment.

The steps I took seem to fix it, I successfully scanned duplicates for 3 users (including the one I tried initially), but now it hung up on a different user (on the Duplicates for user "USER" are: line). I'm trying to scan that user again, hopefully it doesn't hang up again.

Mazvy commented 3 years ago

So, it's still borked. It successfully scanned a handful of users but then it stopped working.

I've let the duplicate command run for ~40 minutes on one user that has less than 3k files. Surely it should've finished by now, a user with ~20K files finished in under 10 minutes. I'm stuck at the same line Duplicates for user "USER" are:

sc

CPU usage is high (#27) but nothing is happening: