Data not fully backed up. (both removing and adding files/folders)

[bluet]

Make sure you are running the latest version of jotta-cli before reporting an issue.

jotta-cli release (jotta-cli version):

root@jottacloud:/src# jotta-cli version                                                                                                                          
-------------------------------------------                                                                                                                      
jottad executable : /usr/bin/jottad                                                                                                                              
jottad appdata    : /var/lib/jottad                                                                                                                              
jottad logfile    : /var/lib/jottad/jottabackup.log                                                                                                              
jottad version    : 0.10.44059                                                                                                                                   
jotta-cli version : 0.10.44059                                                                                                                                   
-------------------------------------------                                                                                                                      
OK                                                                                                                                                               

Description of problem:

Data not fully synced, even with client restarted and waited for hours.
I can see partial scan requested at xxx msgs when adding/removing files/folders, but nothing happen after that.
Even after console shows full scan finished, data is still not showing in jottacloud web.

For example:

• Deleted a folder from local, but still appear in Jottacloud. (scaninterval set to 1h. checked jottacloud web dashboard 24hrs after the adding.)
• Adding a folder and file inside, but not shown in jottacloud web dashboard.

I can hear jottad is generating non-stop noise from my disks (scheduled scanning I thing), and keep generating network traffic to 185.179.129.29:https (this IP is owned by jottacloud) at a rate between 200Kb/s ~ 5Mb/s.

Expected: Data synced fully.

jotta-cli status (jotta-cli status):

root@jottacloud:/src# jotta-cli status                                                                                                                           
------------------------------------------------------------------------------                                                                                   
 Account   : bluetraveler@gmail.com on Jottacloud                                                                                                                
 Usage     : 486.18GiB / ( Unlimited )                                                                                                                           
 Device    : SolidRock                                                                                                                                           
------------------------------------------------------------------------------                                                                                   
 Backups   :                                                                                                                                                     
------------------------------------------------------------------------------                                                                                   
   Path      : /backup/homes                                                                                                                                     
   Files     : 256403 (365.08GiB)                                                                                                                                
   Status    : Transferring - 250756 files (184.91GiB) remaining                                                                                                 

------------------------------------------------------------------------------                                                                                   
OK

Relevant logs for the issue (~/.jottad/jottabackup.log or /var/lib/jotta/jottabackup.log )

pid:340 2021/05/21 13:46:38 Starting /usr/bin/jottad version 0.10.44059 as 0(root)/0 (linux:amd64)                                                                                                                    
pid:340 2021/05/21 13:46:38 jottad.init                                                                                                                                                                               
pid:340 2021/05/21 13:46:38 jottad.init.appdata => /var/lib/jottad                                                                                                                                                    
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad                                                                                                                                                       
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/.ignore (1621565311|4804)                                                                                                                             
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/cldb (1621575792|254582784)                                                                                                                           
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/db (1621574526|125906944)                                                                                                                             
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/downloads                                                                                                                                             
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/downloads/history (1620823185|16384)                                                                                                                  
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/jottabackup-2021-05-20T19-36-52.376.log.gz (1621539424|1855909)                                                                                       
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/jottabackup-2021-05-20T20-16-00.588.log.gz (1621541766|1328609)                                                                                       
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/jottabackup-2021-05-20T20-55-05.066.log.gz (1621544108|1524567)                                                                                       
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/jottabackup.log (1621575998|53665331)                                                                                                                 
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/localfiles (1621574054|118587392)                                                                                                                     
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/nstore - (cd906c5c0735c5ebc5a6e55eb3cdd54c|1621575528|65536)                                                                                          
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/obj                                                                                                                                                   
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/obj/5b296082 (1621575676|830)                                                                                                                         
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/queue.2.q (1621574526|162570240)                                                                                                                      
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/settings.dat (1621570550|208)                                                                                                                         
pid:340 2021/05/21 13:46:38 jottad.appdata      /var/lib/jottad/store - (f5f60ba66ff7257c4783956b3893afd6|1621570547|254)                                                                                             
pid:340 2021/05/21 13:46:38 Loading settings from /var/lib/jottad/settings.dat                                                                                                                                        
pid:340 2021/05/21 13:46:38 jottad.init.cfg.downloadrate => 0B/s                                                                                                                                                      
pid:340 2021/05/21 13:46:38 jottad.init.cfg.uploadrate => 0B/s                                                                                                                                                        
pid:340 2021/05/21 13:46:38 jottad.init.user => [609bb31fffef4e01340d1a25] on [SolidRock] @ [d41d8cd98f00b204e9800998ecf8427e]                                                                                        
pid:340 2021/05/21 13:46:38 jottad.init.cfg.scaninterval =>  0s                                                                                                                                                       
pid:340 2021/05/21 13:46:38 jottad.init.done => ok                                                                                                                                                                    
pid:340 2021/05/21 13:46:38 jottad.startup                                                                                                                                                                            
pid:340 2021/05/21 13:46:38 api.grpc.tls.active => 127.0.0.1:14443                                                                                                                                                    
pid:340 2021/05/21 13:46:38 jottad.startup.update-userinfo => bluetraveler@gmail.com Jottacloud d41d8cd98f00b204e9800998ecf8427e                                                                                      
pid:340 2021/05/21 13:46:40 jottad.startup.confirm-remote => ok                                                                                                                                                       
pid:340 2021/05/21 13:46:40 jottad.startup => ok                                                                                                                                                                      
pid:340 2021/05/21 13:46:41 running 4 delayed startup tasks                                                                                                                                                           
pid:340 2021/05/21 13:46:41 jottad.init.loadedbackup /backup/homes => homes                                                                                                                                           
pid:340 2021/05/21 13:46:41 GET https://repo.jotta.us/archives/VERSION 200                                                                                                                                            
pid:340 2021/05/21 13:46:43 unable to verify root of backup[/backup/homes] => [/backup/homes] is already added to backup                                                                                              
pid:340 2021/05/21 13:46:43 cfg.update.scaninterval 0 => 1h                                                                                                                                                           
pid:340 2021/05/21 13:51:29 fsw: watch  /backup/homes                                                                                                                                                                 
pid:340 2021/05/21 13:51:29 Starting backup events at /backup/homes                                                                                                                                                   
pid:340 2021/05/21 13:51:29 Started scheduler for /backup/homes                                                                                                                                                       
pid:340 2021/05/21 13:51:29 full scan requested at /backup/homes                                                                                                                                                      
pid:340 2021/05/21 13:51:29 Starting full scan at /backup/homes                                                                                                                                                       
pid:340 2021/05/21 13:51:29 ok stat with 0 got filesystem info id[b3bb30801dd5b1b6] type[btrfs] for path /backup/homes                                                                                                
pid:340 2021/05/21 13:51:29 Running full scan of [b3bb30801dd5b1b6/b3bb30801dd5b1b6] /backup/homes                                                                                                                    
pid:340 2021/05/21 13:51:30 Adding group homes
pid:340 2021/05/21 13:51:39 Queueing scheduled scan of /backup/homes                                                                                                                                                  
pid:340 2021/05/21 13:52:49 fsw: watch  /backup/homes                                                                                                                                                                 
pid:340 2021/05/21 14:51:39 Queueing scheduled scan of /backup/homes                                                                                                                                                  
pid:340 2021/05/21 14:52:28 full scan of /backup/homes completed in 1h0m58.524106948s. Found 256403 files 365.08GiB                                                                                                   
pid:340 2021/05/21 15:46:41 GET https://repo.jotta.us/archives/VERSION 200                                                                                                                                            
pid:340 2021/05/21 15:51:39 Queueing scheduled scan of /backup/homes                                                                                                                                                  
pid:340 2021/05/21 16:51:39 Queueing scheduled scan of /backup/homes                                                                                                                                                  
pid:340 2021/05/21 17:46:41 GET https://repo.jotta.us/archives/VERSION 200                                                                                                                                            
pid:340 2021/05/21 17:51:39 Queueing scheduled scan of /backup/homes                                                                                                                                                  
pid:340 2021/05/21 18:51:39 Queueing scheduled scan of /backup/homes                                                                                                                                                  
pid:340 2021/05/21 19:46:41 GET https://repo.jotta.us/archives/VERSION 200                                                                                                                                            
pid:340 2021/05/21 19:51:39 Queueing scheduled scan of /backup/homes                                                                                                                                                  
pid:340 2021/05/21 20:51:39 Queueing scheduled scan of /backup/homes                                                                                                                                                  
pid:340 2021/05/21 21:36:37 partial scan requested at /backup/homes/bluet                                                                                                                                             
pid:340 2021/05/21 21:46:41 GET https://repo.jotta.us/archives/VERSION 200                                                                                                                                            
pid:340 2021/05/21 21:50:53 partial scan requested at /backup/homes                                                                                                                                                   
pid:340 2021/05/21 21:51:39 Queueing scheduled scan of /backup/homes                                                                                                                                                  
pid:340 2021/05/21 21:52:02 partial scan requested at /backup/homes                                                                                                                                                   
pid:340 2021/05/21 21:52:46 partial scan requested at /backup/homes                                                                                                                                                   
pid:340 2021/05/21 21:53:05 partial scan requested at /backup/homes                                                                                                                                                   
pid:340 2021/05/21 21:55:30 partial scan requested at /backup/homes                                                                                                                                                   
pid:340 2021/05/21 22:02:32 partial scan requested at /backup/homes/@eaDir/SYNO@.fileindexdb                                                                                                                          
pid:340 2021/05/21 22:06:01 partial scan requested at /backup/homes                                                                                                                                                   
pid:340 2021/05/21 22:06:13 partial scan requested at /backup/homes                                                                                                                                                   
pid:340 2021/05/21 22:12:32 partial scan requested at /backup/homes/@eaDir/SYNO@.fileindexdb

Traceback

Additional info:

In the screenshot, you can see dejadup and dupli are not in my local side (been removed yesterday), but still showing on jottacloud dashboard.

(never mind #recycle and .ssh. #recycle is in my ignore list, and .ssh is due to read permission.)

[bluet]

Adding folder/file at local, doesn't show in jottacloud too.

I think the most important thing of jotta is to backup my data, but apparently it's not working as expected.

P.S. My whole data size is just few hundred of GBs, so it shouldn't be relate to hitting upload speed limit. (if I understand correctly)

[coopersimon]

Hi, you can try jotta-cli scan --repair [backup name] for the broken backup folders and see if that fixes the issue.

[bluet]

@coopersimon I see. Thanks for your reply. I end up with delete the whole device, logout jotta-cli, and did a full upload again... It took days as the upload speed seems to be limits at ~6MB/s, dunno why.

I'll try jotta-cli scan --repair next time. But what's the backup name I should use? (or where can I find it?) Is it the device name ?

[coopersimon]

It would be homes in your case, but you can also just do jotta-cli scan --repair to repair everything.

[bluet]

@coopersimon today I try to use jotta-cli scan --repair but the command hangs there. It did not print anything, nor returning to bash.
At the same time, jottad outputs pid:434 2021/06/12 00:27:47 repair requested at /backup/homes to logfile, but nothing more after that.

I also tried to check it's status with jotta-cli status in another terminal, but the command also hangs there, without printing anything nor returning to bash.

Am I handling it worng?

jotta-cli version:

root@jottacloud:/src# jotta-cli version
-------------------------------------------
jottad executable : /usr/bin/jottad
jottad appdata    : /var/lib/jottad
jottad logfile    : /var/lib/jottad/jottabackup.log
jottad version    : 0.11.44593
jotta-cli version : 0.11.44593
-------------------------------------------
OK

[coopersimon]

Sounds like it got stuck somewhere - you did everything correctly. I'll investigate.