search

nextcloud / integration_dropbox

🧊 Dropbox integration into Nextcloud
GNU Affero General Public License v3.0
24 stars 10 forks source link

Connects to dropbox but doesn't sync #3

Closed smufa closed 3 years ago

smufa commented 3 years ago

It connects no problem but when I press Import it stays at 0% and 0 files, and only says Dropbox import process will begin soon. I've left it to run for a day but it's still at 0.

julien-nc commented 3 years ago

Is "Cron" selected as the background job method in your Nextcloud admin settings?

The import process is done by background jobs (like the News app). If no job is launched, no import :grin:.

agentff6600 commented 3 years ago

I face a similar problem. Cron is selected. First time it imported quite few files but then stopped. After cancel and restart some files were imported but it says ”process starts soon” now.

Are there any limitations for migration? File size? Amount of files per import attempt?

julien-nc commented 3 years ago

@agentff6600 Thanks for the feedback. For now, once a job has downloaded more than 500 MB, it stops and schedules another job for next time cron.php is launched. Next job will go on importing at least 500 MB and so on.

If there is no error, stopping and starting the import won't make any difference. Could you check nextcloud.log to see if there is any related error?

agentff6600 commented 3 years ago

@eneiluj thanks!

I restarted the import again to see if errors appear. It says "Importing dropbox files for ncp" like 6 times but no additional infos. In the intergration menu it says that it imported an amount of file on a certain time that is inbetween the error messages.

What if a file is bigger then 500 MB?

julien-nc commented 3 years ago

What if a file is bigger then 500 MB?

The condition is checked between each files so it will work fine with files bigger than 500 MB.

So now you can wait until the background job has been launched a few times, if the number of downloaded files does not increase, you have a problem :grin:. Otherwise everything was fine after all. Let's see.

agentff6600 commented 3 years ago

got this error:

core] Error: OCP\Files\NotFoundException:  at <<closure>>

0. /var/www/nextcloud/lib/private/Files/Node/Node.php line 231
   OC\Files\Node\Node->getFileInfo()
1. /var/www/nextcloud/lib/private/Files/Node/Node.php line 129
   OC\Files\Node\Node->getPermissions()
2. /var/www/nextcloud/lib/private/Files/Node/File.php line 113
   OC\Files\Node\Node->checkPermissions(3)
3. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 245
   OC\Files\Node\File->fopen("w")
4. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 192
   OCA\Dropbox\Service\DropboxStorageAPIService->getFile("sl.Ao08lxAGxYEY ... U", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", “USERNAME”, {.tag: "file",na ... "}, OC\Files\Node\Folder {})
5. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 123
   OCA\Dropbox\Service\DropboxStorageAPIService->importFiles("sl.Ao08lxAGxYEY ... U", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", "ncp", "Dropbox import", 500000000, 2710)
6. /var/www/nextcloud/apps/integration_dropbox/lib/BackgroundJob/ImportDropboxJob.php line 36
   OCA\Dropbox\Service\DropboxStorageAPIService->importDropboxJob(“USERNAME”)
7. /var/www/nextcloud/lib/public/BackgroundJob/Job.php line 80
   OCA\Dropbox\BackgroundJob\ImportDropboxJob->run({user_id: “USERNAME”})
8. /var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php line 48
   OCP\BackgroundJob\Job->execute(OC\BackgroundJob\JobList {}, OC\Log {})
9. /var/www/nextcloud/cron.php line 127
   OCP\BackgroundJob\QueuedJob->execute(OC\BackgroundJob\JobList {}, OC\Log {})

at 2021-01-06T14:15:40+00:00
julien-nc commented 3 years ago

Those storage APIs are bringing so many problems :grin:

This one might be solved in v0.0.14-nightly that was just released. Could you try it out? It would be better to delete the import directory before launching the import process to make sure we see this "problematic" (don't know why) file again.

Thanks for the feedback!

agentff6600 commented 3 years ago

Yay! Thanks for the quick response! As much as I would like to help you I am a bit of a noob when it comes to applying nightly builds. Should I switch to the beta channel or is there another way?

julien-nc commented 3 years ago

Well it's easier if I just publish and you update as you are used to. Anyway this change is positive or at worst has no effect. v0.0.14 is out. Just update as usual.

Using nightly builds requires to replace the app directory with the one downloaded from the appstore. Then when you want to go back to official releases, you just delete the app directory and "download and install" again from Nextcloud apps settings.

agentff6600 commented 3 years ago

So no serious errors so far. Cron jobs are executed every 15min and files are imported in a steady pace. Thanks for the work and the swift update!

agentff6600 commented 3 years ago

Hi again!

it worked for quite a while now and then two errors showed up:

[cron] Error: Error: Call to undefined method OCA\Dropbox\Service\DropboxAPIService::fileRequest() at <<closure>>

0. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 256
   OCA\Dropbox\Service\DropboxAPIService->downloadFile("sl.Ao5j14iUmffb ... E", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", “USER”, null, "id:Lfn3lFzGqD8AAAAAAAABiA")
1. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 193
   OCA\Dropbox\Service\DropboxStorageAPIService->getFile("sl.Ao3Si-IMOLfO ... k", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", "USER", {.tag: "file",na ... "}, OC\Files\Node\Folder {})
2. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 124
   OCA\Dropbox\Service\DropboxStorageAPIService->importFiles("sl.Ao3Si-IMOLfO ... k", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", "USER", "Dropbox import", 500000000, 1368)
3. /var/www/nextcloud/apps/integration_dropbox/lib/BackgroundJob/ImportDropboxJob.php line 36
   OCA\Dropbox\Service\DropboxStorageAPIService->importDropboxJob("USER")
4. /var/www/nextcloud/lib/public/BackgroundJob/Job.php line 80
   OCA\Dropbox\BackgroundJob\ImportDropboxJob->run({user_id: “USER”})
5. /var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php line 48
   OCP\BackgroundJob\Job->execute(OC\BackgroundJob\JobList {}, OC\Log {})
6. /var/www/nextcloud/cron.php line 127
   OCP\BackgroundJob\QueuedJob->execute(OC\BackgroundJob\JobList {}, OC\Log {})

at 2021-01-07T12:45:31+00:00
[index] Error: TypeError: Argument 10 passed to OCA\Dropbox\Controller\DropboxAPIController::__construct() must be of the type string, null given at <<closure>>

0. <<closure>>
   OCA\Dropbox\Controller\DropboxAPIController->__construct("integration_dropbox", OC\AppFramework\Http\Request {}, OC\Server {}, OC\AllConfig {}, OC\L10N\LazyL10N {}, OC\App\AppManager {}, OC\Files\AppData\AppData {}, OC\AppFramework\ScopedPsrLogger {}, OCA\Dropbox\Serv ... }}, null)
1. /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php line 101
   ReflectionClass->newInstanceArgs(["integration_dr ... l])
2. /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php line 109
   OC\AppFramework\Utility\SimpleContainer->buildClass(ReflectionClass  ... "})
3. /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php line 126
   OC\AppFramework\Utility\SimpleContainer->resolve("OCA\\Dropbox\\C ... r")
4. /var/www/nextcloud/lib/private/AppFramework/DependencyInjection/DIContainer.php line 455
   OC\AppFramework\Utility\SimpleContainer->query("OCA\\Dropbox\\C ... r")
5. /var/www/nextcloud/lib/private/AppFramework/DependencyInjection/DIContainer.php line 427
   OC\AppFramework\DependencyInjection\DIContainer->queryNoFallback("OCA\\Dropbox\\C ... r")
6. /var/www/nextcloud/lib/private/AppFramework/App.php line 139
   OC\AppFramework\DependencyInjection\DIContainer->query("OCA\\Dropbox\\C ... r")
7. /var/www/nextcloud/lib/private/Route/Router.php line 308
   OC\AppFramework\App::main("OCA\\Dropbox\\C ... r", "getImportDropboxInformation", OC\AppFramework\ ... {}, {action: null,_r ... "})
8. /var/www/nextcloud/lib/base.php line 1008
   OC\Route\Router->match("/apps/integrati ... o")
9. /var/www/nextcloud/index.php line 37
   OC::handleRequest()

GET /apps/integration_dropbox/import-files-info
from 192.168.1.1 at 2021-01-07T14:41:20+00:00
julien-nc commented 3 years ago

Oh sorry about that. Stupid mistakes. Those 2 should be fixed in v0.0.15 which is out.

agentff6600 commented 3 years ago

Hi again!

have two more hick-ups to report.

One is because a file was a corrupted image (which I deleted on dropbox now):

[core] Error: OCP\Lock\LockedException: "Dropbox import/Camera Uploads/2013-12-23 10.15.31.png" is locked, existing lock on file: exclusive at <<closure>>

 0. /var/www/nextcloud/lib/private/Files/View.php line 2074
    OC\Files\View->lockPath("/ncp/files/Drop ... g", 1, false)
 1. /var/www/nextcloud/lib/private/Files/View.php line 1147
    OC\Files\View->lockFile("/ncp/files/Drop ... g", 1)
 2. /var/www/nextcloud/lib/private/Files/View.php line 724
    OC\Files\View->basicOperation("unlink", "/ncp/files/Drop ... g", ["delete"])
 3. /var/www/nextcloud/lib/private/Files/Node/File.php line 132
    OC\Files\View->unlink("/ncp/files/Drop ... g")
 4. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 265
    OC\Files\Node\File->delete()
 5. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 193
    OCA\Dropbox\Service\DropboxStorageAPIService->getFile("sl.Ao6JpcbFIEvo ... s", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", "USER", {.tag: "file",na ... "}, OC\Files\Node\Folder {})
 6. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 124
    OCA\Dropbox\Service\DropboxStorageAPIService->importFiles("sl.Ao6JpcbFIEvo ... s", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", "USER", "Dropbox import", 500000000, 8360)
 7. /var/www/nextcloud/apps/integration_dropbox/lib/BackgroundJob/ImportDropboxJob.php line 36
    OCA\Dropbox\Service\DropboxStorageAPIService->importDropboxJob("USER")
 8. /var/www/nextcloud/lib/public/BackgroundJob/Job.php line 80
    OCA\Dropbox\BackgroundJob\ImportDropboxJob->run({user_id: "USER"})
 9. /var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php line 48
    OCP\BackgroundJob\Job->execute(OC\BackgroundJob\JobList {}, OC\Log {})
10. /var/www/nextcloud/cron.php line 127
    OCP\BackgroundJob\QueuedJob->execute(OC\BackgroundJob\JobList {}, OC\Log {})

at 2021-01-08T06:17:16+00:00

and this one popped up after I reinitiated the import again (maybe I should have done that?):

[cron] Error: TypeError: Return value of OCA\Dropbox\Service\DropboxStorageAPIService::getFile() must be of the type int or null, float returned at <<closure>>

0. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 193
   OCA\Dropbox\Service\DropboxStorageAPIService->getFile("sl.Ao6JpcbFIEvo ... s", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", "USER", {.tag: "file",na ... "}, OC\Files\Node\Folder {})
1. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 124
   OCA\Dropbox\Service\DropboxStorageAPIService->importFiles("sl.Ao6JpcbFIEvo ... s", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", "USER", "Dropbox import", 500000000, 179)
2. /var/www/nextcloud/apps/integration_dropbox/lib/BackgroundJob/ImportDropboxJob.php line 36
   OCA\Dropbox\Service\DropboxStorageAPIService->importDropboxJob("USER")
3. /var/www/nextcloud/lib/public/BackgroundJob/Job.php line 80
   OCA\Dropbox\BackgroundJob\ImportDropboxJob->run({user_id: "USER"})
4. /var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php line 48
   OCP\BackgroundJob\Job->execute(OC\BackgroundJob\JobList {}, OC\Log {})
5. /var/www/nextcloud/cron.php line 127
   OCP\BackgroundJob\QueuedJob->execute(OC\BackgroundJob\JobList {}, OC\Log {})

at 2021-01-08T09:22:41+00:00
julien-nc commented 3 years ago

Thanks for the feedback. v0.0.16 is out, fixing those 2 issues.

a file was a corrupted image

Hmmm this shouldn't cause problems when downloading the file. I don't know why some files are locked. It also happens with google integration app. I can avoid the crash but it would be nicer to understand the problem and properly fix it :grin:.

agentff6600 commented 3 years ago

Thanks a bunch @eneiluj !

How can I help?

agentff6600 commented 3 years ago

So, the script had a good run! Up until this morning:

[core] Error: OCP\Files\ForbiddenException: Invalid path at <<closure>>

 0. /var/www/nextcloud/lib/private/Files/Storage/Local.php line 236
    OC\Files\Storage\Local->getSourcePath("files/Dropbox import/Public/.htaccess")
 1. /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 226
    OC\Files\Storage\Local->file_exists("files/Dropbox import/Public/.htaccess")
 2. /var/www/nextcloud/lib/private/Files/View.php line 1340
    OC\Files\Storage\Wrapper\Wrapper->file_exists("files/Dropbox import/Public/.htaccess")
 3. /var/www/nextcloud/lib/private/Files/View.php line 1389
    OC\Files\View->getCacheEntry(OCA\Files_Trashb ... }}, "files/Dropbox import/Public/.htaccess", "/ncp/files/Drop ... s")
 4. /var/www/nextcloud/lib/private/Files/Node/Root.php line 201
    OC\Files\View->getFileInfo("/ncp/files/Drop ... s")
 5. /var/www/nextcloud/lib/private/Files/Node/Folder.php line 140
    OC\Files\Node\Root->get("/ncp/files/Drop ... s")
 6. /var/www/nextcloud/lib/private/Files/Node/Folder.php line 149
    OC\Files\Node\Folder->get(".htaccess")
 7. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 244
    OC\Files\Node\Folder->nodeExists(".htaccess")
 8. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 193
    OCA\Dropbox\Service\DropboxStorageAPIService->getFile("sl.ApKDyWAXuKWn ... I", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", "ncp", {.tag: "file",na ... "}, OC\Files\Node\Folder {})
 9. /var/www/nextcloud/apps/integration_dropbox/lib/Service/DropboxStorageAPIService.php line 124
    OCA\Dropbox\Service\DropboxStorageAPIService->importFiles("sl.ApKDyWAXuKWn ... I", "3PWgkpmFGJ4AAAA ... D", "hh276c3kzellh2x", "rdsuw9qg4y4fj5p", "ncp", "Dropbox import", 500000000, 17155)
10. /var/www/nextcloud/apps/integration_dropbox/lib/BackgroundJob/ImportDropboxJob.php line 36
    OCA\Dropbox\Service\DropboxStorageAPIService->importDropboxJob("ncp")
11. /var/www/nextcloud/lib/public/BackgroundJob/Job.php line 80
    OCA\Dropbox\BackgroundJob\ImportDropboxJob->run({user_id: "ncp"})
12. /var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php line 48
    OCP\BackgroundJob\Job->execute(OC\BackgroundJob\JobList {}, OC\Log {})
13. /var/www/nextcloud/cron.php line 127
    OCP\BackgroundJob\QueuedJob->execute(OC\BackgroundJob\JobList {}, OC\Log {})

at 2021-01-12T05:40:26+00:00
agentff6600 commented 3 years ago

@eneiluj I successfully imported two dropbox accounts! got a few warning but seems like everything is in its place. Thanks!

djrpm commented 2 years ago

Hi I have installed dropbox integration version 1.0.4, activated the cron but it has always been with me

"0 files imported Dropbox import process will begin soon "

in the logs I get out

"[integration_dropbox] Error: Importing dropbox files for djrpm

at 2022-05-19T17: 25: 03 + 00: 00 "

what can I do ?