search

droboports / crashplan

CrashPlan DroboApp build scripts
GNU General Public License v2.0
17 stars 1 forks source link

Drobo 5N 4.2.0 stuck in "Connecting to backup destination" #7

Closed ryanspletzer closed 9 years ago

ryanspletzer commented 9 years ago

Issue on Drobo 5N with 4.2.0. Fresh install with latest Java8 and Locale packages. Log file shows the following:

Exception in thread "W1636663_ScanWrkr" java.lang.NoClassDefFoundError: Could not initialize class com.code42.jna.inotify.InotifyManager
    at com.code42.jna.inotify.JNAInotifyFileWatcherDriver.<init>(JNAInotifyFileWatcherDriver.java:21)
    at com.code42.backup.path.BackupSetsManager.initFileWatcherDriver(BackupSetsManager.java:393)
    at com.code42.backup.path.BackupSetsManager.startScheduledFileQueue(BackupSetsManager.java:331)
    at com.code42.backup.path.BackupSetsManager.access$1600(BackupSetsManager.java:66)
    at com.code42.backup.path.BackupSetsManager$ScanWorker.delay(BackupSetsManager.java:1073)
    at com.code42.utils.AWorker.run(AWorker.java:158)
    at java.lang.Thread.run(Unknown Source)
Exception in thread "W23960285_ScanWrkr" java.lang.NoClassDefFoundError: Could not initialize class com.code42.jna.inotify.InotifyManager
    at com.code42.jna.inotify.JNAInotifyFileWatcherDriver.<init>(JNAInotifyFileWatcherDriver.java:21)
    at com.code42.backup.path.BackupSetsManager.initFileWatcherDriver(BackupSetsManager.java:393)
    at com.code42.backup.path.BackupSetsManager.startScheduledFileQueue(BackupSetsManager.java:331)
    at com.code42.backup.path.BackupSetsManager.access$1600(BackupSetsManager.java:66)
    at com.code42.backup.path.BackupSetsManager$ScanWorker.delay(BackupSetsManager.java:1073)
    at com.code42.utils.AWorker.run(AWorker.java:158)
    at java.lang.Thread.run(Unknown Source)
ricardopadilha commented 9 years ago

Please reinstall CrashPlan. Some files are missing.

ryanspletzer commented 9 years ago

Re-installed the latest crashplan.tgz 4.2.0 release from this repo, but even before I started backing up I saw the error in log.txt... I think com.code42.jna.inotify.InotifyManager is something missing in the core CrashPlan package. Is there something else I can look at in the directory structure that would provide more info as to what's wrong / missing?

ricardopadilha commented 9 years ago

Sorry, I can't reproduce your problem. I just installed locale, then java8, then crashplan on a freshly reset 5N and it started just fine. The only advice I can give you is to remove locale, java8, and crashplan, restart the Drobo and start over.

By the way, are you sure you are using the proper release? The DroboFS version might not work on the Drobo5N.

deekayen commented 9 years ago

I'm having the same problem. I removed Locale, Crashplan, and Java completely, then restored without even my backupArchives directory. I'm not even sure where to find logs to help with diagnosing it. I haven't been able to get my Drobo 5N to backup to Crashplan Central or my local computers to connect to my Crashplan service on the Drobo.

smenzer commented 9 years ago

This page describes a possible solution to this problem: https://randomwindowstips.wordpress.com/2013/02/25/crashplan-pro-for-linux-stuck-at-waiting-for-backup-or-connecting-to-backup-destination/

deekayen commented 9 years ago

@smenzer so did that work for you? did you use /tmp/DroboApps/crashplan or something else? I set my run.conf to SRV_JAVA_OPTS="-Djava.io.tmpdir=/tmp/DroboApps/crashplan ... and had no improvement.

smenzer commented 9 years ago

i used /var/crashplan_tmp, did a chmod 777 on it (i know not secure, but trying everything), and also did a chown Admin:Admin (where Admin is the Drobo user account). it seemed to fix that error message in my logs, but I'm still having issues getting crashplan to properly run backups on the drobo...still just says "Connecting to backup destination", but I can't find any errors anywhere

ricardopadilha commented 9 years ago

Please do not use /var for this purpose. /var only has a few hundred kb free, and if it gets full your Drobo will not be able to boot anymore.

By the way, both versions (FS and 5N) already define a java.io.tmpdir. Check https://github.com/droboports/crashplan/blob/master/src/dest/service.sh#L42 and https://github.com/droboports/crashplan/blob/drobofs/src/dest/service.sh#L42.

smenzer commented 9 years ago

hi @ricardopadilha I do see that java.io.tmpdir is set within service.sh. However, when I unset it in run.conf then I'm still getting that error and my backups to the Drobo stop working (same errors). Any thoughts?

deekayen commented 9 years ago

I started to work on my Drobo 5N again tonight to figure this out and noticed my laptop was backing up to my Drobo's Crashplan service on my LAN and the Drobo 5N is talking to Crashplan Central. It all appears to be working!

The only change I made after doing a complete re-install of Java8, Locale, and Crashplan 4.2.0 was to add the tmpdir to the start of my run.conf like SRV_JAVA_OPTS="-Djava.io.tmpdir=/tmp/DroboApps/crashplan ..., reboot the Drobo, and waited 2 days.

I'd rather have the tmpdir under /tmp/DroboFS so I don't accidentally fill up my /tmp partition. Anyone have a recommendation?

Lammerink commented 9 years ago

I'm having the same issue, I got Crashplan to load but it's stuck "Waiting for Backup". It seems like it's connecting to the CrashPlan server (green bubble) but it's just not backing up, Inbound backups connect but say "Backup disabled".

I already tried to define a java.io.tmpdir, but this didn't resolve the issue.

I uninstalled Crashplan, locale and Java8. Rebooted the Drobo 5N, installed the latest packages of Locale and Java8 from Github. Rebooted again, then installed crashplan.

The only change I made to the Crashplan package was to set the Max Java Heap size to 1152mb.

I went through the logfiles and found this:

05.25.15 10:59:14.282 WARN    main                 com.code42.utils.UniqueId               ] critical error converting IP address into int

don't know if this has anything to do with it but maybe it will.

Here is the whole block and context.

[05.25.15 10:59:13.855 INFO    main                 root                                    ] Locale changed to English
[05.25.15 10:59:13.860 INFO    main                 root                                    ] *************************************************************
[05.25.15 10:59:13.861 INFO    main                 root                                    ] *************************************************************
[05.25.15 10:59:13.861 INFO    main                 root                                    ] STARTED CrashPlanService
[05.25.15 10:59:13.867 INFO    main                 root                                    ] CPVERSION = 4.2.0 - 1425276000420 (2015-03-02T06:00:00:420+0000) - Build: 61
[05.25.15 10:59:13.868 INFO    main                 root                                    ] LOCALE = English
[05.25.15 10:59:13.871 INFO    main                 root                                    ] ARGS = [  ]
[05.25.15 10:59:13.872 INFO    main                 root                                    ] *************************************************************
[05.25.15 10:59:14.206 INFO    main                 root                                    ] Adding shutdown hook.
[05.25.15 10:59:14.229 INFO    main                 root                                    ] BEGIN Loading Configuration
[05.25.15 10:59:14.282 WARN    main                 com.code42.utils.UniqueId               ] critical error converting IP address into int
[05.25.15 10:59:14.426 INFO    main                 root                                    ] BEGIN Copy Custom
[05.25.15 10:59:14.427 INFO    main                 root                                    ]   Directories: [.Custom, custom, /mnt/DroboFS/Shares/DroboApps/crashplan/app/conf/.Custom, /mnt/DroboFS/Shares/DroboApps/crashplan/app/conf/custom]
[05.25.15 10:59:14.427 INFO    main                 root                                    ]   NOT waiting for custom skin to appear
[05.25.15 10:59:14.428 INFO    main                 root                                    ]   NO customizations found.
[05.25.15 10:59:14.429 INFO    main                 root                                    ] END Copy Custom
[05.25.15 10:59:14.432 INFO    main                 root                                    ] CpsFoldersMigrate is not necessary. /mnt/DroboFS/Shares/DroboApps/crashplan/app/conf/my.service.xml file does not exists.
[05.25.15 10:59:14.445 INFO    main                 root                                    ]   Loading from default: /mnt/DroboFS/Shares/DroboApps/crashplan/app/conf/default.service.xml
[05.25.15 10:59:14.880 INFO    main                 root                                    ]   Loading ServiceConfig, newInstall=true, version=6, configDateMs=null, installVersion=1425276000420
[05.25.15 10:59:14.922 INFO    main                 root                                    ]   OS = Linux
[05.25.15 10:59:14.927 INFO    main                 root                                    ]   Initializing backup paths last modified to now. lastModified=1
[05.25.15 10:59:15.258 INFO    main                 root                                    ]   AuthorityLocation@26694800[ location=central.crashplan.com:443, hideAddress=false ]
[05.25.15 10:59:15.264 INFO    main                 root                                    ]   Checking Java memory heap max.
[05.25.15 10:59:15.279 INFO    main                 root                                    ]     Save Java memory max heap as 1152m
[05.25.15 10:59:15.406 INFO    main                 root                                    ] END Loading Configuration
jtux Loaded.
Exception in thread "W568448_ScanWrkr" java.lang.NoClassDefFoundError: Could not initialize class com.code42.jna.inotify.InotifyManager
    at com.code42.jna.inotify.JNAInotifyFileWatcherDriver.<init>(JNAInotifyFileWatcherDriver.java:21)
    at com.code42.backup.path.BackupSetsManager.initFileWatcherDriver(BackupSetsManager.java:393)
    at com.code42.backup.path.BackupSetsManager.startScheduledFileQueue(BackupSetsManager.java:331)
    at com.code42.backup.path.BackupSetsManager.access$1600(BackupSetsManager.java:66)
    at com.code42.backup.path.BackupSetsManager$ScanWorker.delay(BackupSetsManager.java:1073)
    at com.code42.utils.AWorker.run(AWorker.java:158)
    at java.lang.Thread.run(Unknown Source)
2015-05-25 04-08-13: ./crashplan/service.sh stop
ricardopadilha commented 9 years ago

Could you guys please post the output of stat /mnt/DroboFS/Shares/DroboApps/crashplan/tmp ?

It should show something like this:

  File: /mnt/DroboFS/Shares/DroboApps/crashplan/tmp
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: 801h/2049d  Inode: 66077760    Links: 2
Access: (0775/drwxrwxr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-10-27 17:13:23.000000000
Modify: 2014-10-27 17:13:23.000000000
Change: 2014-10-27 17:17:57.000000000
Lammerink commented 9 years ago

This is my output:

stat /mnt/DroboFS/Shares/DroboApps/crashplan/tmp
  File: /mnt/DroboFS/Shares/DroboApps/crashplan/tmp
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: 801h/2049d  Inode: 34046149    Links: 3
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-05-25 03:59:12.000000000
Modify: 2015-05-25 04:06:10.000000000
Change: 2015-05-25 04:06:10.000000000
ricardopadilha commented 9 years ago

Interesting. Do you mind doing this: chmod 775 /mnt/DroboFS/Shares/DroboApps/crashplan/tmp on your Drobo, and making sure that your java.io.tmpdir is as it was originally defined in service.sh (i.e., to /mnt/DroboFS/Shares/DroboApps/crashplan/tmp)?

This page seems to suggest that the wrong permissions on the tmp folder are the cause of this problem.

deekayen commented 9 years ago

I removed the hard-coded tmp dir in my run.conf (setting it back to default setup settings) and it went back to a broken state after a reboot of the Drobo.

# pwd
/mnt/DroboFS/Shares/DroboApps/crashplan
# stat tmp
  File: tmp
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: 801h/2049d  Inode: 57147668    Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-05-22 08:52:58.000000000
Modify: 2015-05-22 08:52:58.000000000
Change: 2015-05-22 08:52:58.000000000

# ls tmp
# ls /tmp/DroboApps/crashplan/
app.log                framework_version.txt  pid.txt
backup_files.log.0     history.log.0          restore_files.log.0
depends.txt            log.txt                service.log.0
description.txt        name.txt               version.txt

I'm going to try setting the /mnt/DroboFS/Shares/DroboApps/crashplan/tmp dir to 775 like @ricardopadilha and see what happens over the next day.

Lammerink commented 9 years ago

I removed the hard-coded java.io.tmpdir did the chmod change and restarted crashplan. No change, Tried again hard-coding in /mnt/DroboFS/Shares/DroboApps/crashplan/tmp as java.io.tmpdir. No luck either.

ricardopadilha commented 9 years ago

Ok, how about chmod 777 /mnt/DroboFS/Shares/DroboApps/crashplan/tmp ?

deekayen commented 9 years ago

Setting /mnt/DroboFS/Shares/DroboApps/crashplan/tmp to 777 hasn't made mine happy yet.

ricardopadilha commented 9 years ago

I managed to reproduce in on my 5N, from a clean install. Investigation under way.

deekayen commented 9 years ago

I checked on the status of my 777 change to /mnt/DroboFS/Shares/DroboApps/crashplan/tmp and found that my laptop was able to backup to my Drobo 5N, but the Drobo 5N is still "Waiting to backup" to Crashplan Central. Crashplan Central is registering that the Drobo has recently contacted them as if it has done a backup, but the local client claims that it hasn't even started.

smenzer commented 9 years ago

I have removed the java.io.tmpdir setting and still get the error reported earlier, however I do see my computers backing up to the Drobo now. For debugging, here's the output requested earlier in case it helps:

# stat /mnt/DroboFS/Shares/DroboApps/crashplan/tmp
  File: "/mnt/DroboFS/Shares/DroboApps/crashplan/tmp"
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: 801h/2049d  Inode: 25719404    Links: 3
Access: (0777/drwxrwxrwx)  Uid: ( 1000/   Admin)   Gid: ( 1000/   Admin)
Access: 2015-05-22 11:54:52.000000000
Modify: 2015-05-22 11:54:56.000000000
Change: 2015-05-24 12:49:47.000000000
# ls -la /mnt/DroboFS/Shares/DroboApps/crashplan/tmp
drwxrwxrwx    3 Admin    Admin        4096 May 22 11:54 .
drwxr-xr-x    7 Admin    Admin        4096 May 22 12:41 ..
drwxrwxrwx    2 Admin    Admin        4096 May 22 11:54 .AppleDouble
smenzer commented 9 years ago

I'm back to getting errors again - I had tried to change the location of the backups on the drobo (from the default location of /mnt/DroboFS/Shares/DroboApps/crashplan/app/backupArchives to another share I created on the drobo /mnt/DroboFS/Shares/Backup/DroboCrashplan. That broke everything, so I switched it all back to the original location and now it's not working. Earlier, when I just let it sit overnight it seemed to work again, so I'm hoping it does the same, but still odd.

When I login to the Drobo via Crashplan's UI (via the port forwarding and such), when I go to the History tab, I see the following for each of the computers trying to backup to the Drobo: Not ready for backup from <Computer>. Reason: The destination is not available

Within /tmp/DroboApps/crashplan/logs.tx I have the same messages that others have posted above - only the error on the java.io.tmpdir and the critical error converting IP address into int warning.

ricardopadilha commented 9 years ago

I think I found the problem. Can you please check your log files for this error message?

[05.27.15 16:32:56.590 ERROR   main                 backup42.service.backup.BackupController] Exception setting up BackupController! java.lang.UnsatisfiedLinkError: jnidispatch (/com/sun/jna/linux-arm/libjnidispatch.so) not found in resource path, java.lang.UnsatisfiedLinkError: jnidispatch (/com/sun/jna/linux-arm/libjnidispatch.so) not found in resource path
java.lang.UnsatisfiedLinkError: jnidispatch (/com/sun/jna/linux-arm/libjnidispatch.so) not found in resource path
    at com.sun.jna.Native.loadNativeLibraryFromJar(Native.java:704)
    at com.sun.jna.Native.loadNativeLibrary(Native.java:681)
    at com.sun.jna.Native.<clinit>(Native.java:109)
    at com.code42.jna.LinuxPlatform.<init>(LinuxPlatform.java:73)
    at com.code42.jna.PlatformFactory.getLinuxPlatform(PlatformFactory.java:41)
    at com.code42.jna.inotify.InotifyManager.<clinit>(InotifyManager.java:45)
    at com.code42.jna.inotify.JNAInotifyFileWatcherDriver.<init>(JNAInotifyFileWatcherDriver.java:21)
    at com.code42.backup.path.BackupSetsManager.initFileWatcherDriver(BackupSetsManager.java:393)
    at com.code42.backup.path.BackupSetsManager.setUp(BackupSetsManager.java:129)
    at com.code42.backup.BackupManager.setUp(BackupManager.java:140)
    at com.backup42.service.backup.BackupController.setUp(BackupController.java:463)
    at com.backup42.service.CPService.start(CPService.java:490)
    at com.backup42.service.CPService.main(CPService.java:1849)
smenzer commented 9 years ago

@ricardopadilha I looked through /tmp/DroboApps/crashplan/logs.txt and couldn't find the string BackupController.

This is still the only error I see, and it gets thrown shortly after a restart and only appears once per restart

Exception in thread "W780592_ScanWrkr" java.lang.NoClassDefFoundError: Could not initialize class com.code42.jna.inotify.InotifyManager
        at com.code42.jna.inotify.JNAInotifyFileWatcherDriver.<init>(JNAInotifyFileWatcherDriver.java:21)
        at com.code42.backup.path.BackupSetsManager.initFileWatcherDriver(BackupSetsManager.java:393)                                                      
        at com.code42.backup.path.BackupSetsManager.startScheduledFileQueue(BackupSetsManager.java:331)            
        at com.code42.backup.path.BackupSetsManager.access$1600(BackupSetsManager.java:66)                               
        at com.code42.backup.path.BackupSetsManager$ScanWorker.delay(BackupSetsManager.java:1073)                                          
        at com.code42.utils.AWorker.run(AWorker.java:158)                                                      
        at java.lang.Thread.run(Unknown Source)
ricardopadilha commented 9 years ago

What about UnsatisfiedLinkError? That is the critical one.

smenzer commented 9 years ago

@ricardopadilha looks like I was looking in the wrong log file. in /tmp/DroboApps/crashplan/service.log.1 I do see both the BackupController text as well as the UnsatisfiedLinkError. They seem to have happened just after restarting the crashplan service, which based on the logs.txt file was at 05.26.15 08:24:10.798

[05.26.15 08:24:24.625 ERROR   main                 backup42.service.backup.BackupController] Exception setting up BackupController! java.lang.UnsatisfiedLinkError: jnidispatch (/com/sun/jna/linux-arm/libjnidispatch.so) not found in resource path, java.lang.UnsatisfiedLinkError: jnidispatch (/com/sun/jna/linux-arm/libjnidispatch.so) not found in resource path
java.lang.UnsatisfiedLinkError: jnidispatch (/com/sun/jna/linux-arm/libjnidispatch.so) not found in resource path
    at com.sun.jna.Native.loadNativeLibraryFromJar(Native.java:704)
    at com.sun.jna.Native.loadNativeLibrary(Native.java:681)
    at com.sun.jna.Native.<clinit>(Native.java:109)
    at com.code42.jna.LinuxPlatform.<init>(LinuxPlatform.java:73)
    at com.code42.jna.PlatformFactory.getLinuxPlatform(PlatformFactory.java:41)
    at com.code42.jna.inotify.InotifyManager.<clinit>(InotifyManager.java:45)
    at com.code42.jna.inotify.JNAInotifyFileWatcherDriver.<init>(JNAInotifyFileWatcherDriver.java:21)
    at com.code42.backup.path.BackupSetsManager.initFileWatcherDriver(BackupSetsManager.java:393)
    at com.code42.backup.path.BackupSetsManager.setUp(BackupSetsManager.java:129)
    at com.code42.backup.BackupManager.setUp(BackupManager.java:140)
    at com.backup42.service.backup.BackupController.setUp(BackupController.java:463)
    at com.backup42.service.CPService.start(CPService.java:490)
    at com.backup42.service.CPService.main(CPService.java:1849)
ricardopadilha commented 9 years ago

I compiled new versions with a fix for the UnsatisfiedLinkError. I deleted the previous tgz files for the 4.2.0 releases. Please re-download and install them.

smenzer commented 9 years ago

@ricardopadilha awesome! just so I don't screw anything up...to install, just drop the new .tgz file in the DroboApps directory and run /bin/sh /usr/bin/DroboApps.sh install right? or are there any other steps involved?

ricardopadilha commented 9 years ago

That is basically it, yes. If you want to be extra safe you can move crashplan/app/backupArchives out of the way, but normally it should not be necessary.

smenzer commented 9 years ago

alright thanks!

I'm getting an error, though. I got this originally and ended up commenting out the lines to get crashplan to startup, but figured you should see this:

2015-05-27 12-16-42: /mnt/DroboFS/Shares/DroboApps/crashplan/service.sh start
/mnt/DroboFS/Shares/DroboApps/crashplan/service.sh: set: line 64: illegal option -o pipefail

/mnt/DroboFS/Shares/DroboApps/crashplan/service.sh:64 is: set -o pipefail # propagate last error code on pipe

I believe there are two other places this shows up as well

ricardopadilha commented 9 years ago

Argh, that was a mindless merge. I'll update the DroboFS version asap.

smenzer commented 9 years ago

thanks for the quick responses today!

ricardopadilha commented 9 years ago

New DroboFS version out, fixed by commit 297f1181d5739ffd99f5cb430851612d71b13511.

smenzer commented 9 years ago

that seems to be working! i'll post back here again in a day or two to confirm.

appreciate it!

ryanspletzer commented 9 years ago

Thanks @ricardopadilha!! I'm going to try this tonight.

Quick Q: when installing the .tgz files, it doesn't make a difference whether you do it as root or as admin, correct? It should work either way?

ricardopadilha commented 9 years ago

Always, always as root.

ryanspletzer commented 9 years ago

Ok, good, I was doing it as root before but wanted to double check. Thanks!

4bitfocus commented 9 years ago

I installed the latest version (from about 10 minutes ago) on my Drobo5N and it seemed to have fixed the issue. CP reports "Backup complete!" after synchronizing blocks so it sounds like it was able to make the connection. Thanks for the effort on this!

ryanspletzer commented 9 years ago

Pulled down the latest Drobo5N locale / java8 / crashplan releases and freshly installed tgz's as root (and rebooted the drobo after each install to be sure), everything is working swimmingly now! Many thanks for the fixes!