yadayada / acd_cli

An unmaintained command line interface and FUSE filesystem for Amazon (Cloud) Drive
Other
1.35k stars 165 forks source link

Root node not found. Please sync...after syncing. #469

Open ebyrne242 opened 8 years ago

ebyrne242 commented 8 years ago

I've been running acdcli on a couple of machines (one of which is also CentOS 7) and things have been working great. Just did a fresh install with pip3 install --upgrade git+https://github.com/yadayada/acd_cli.git on a new machine, and sync appears to complete successfully, but I can't mount. I'm using python34-3.4.3-7.el7.x86_64 on CentOS 7.2.

$ acdcli -v sync
16-11-01 09:47:18.830 [INFO] [acd_cli] - Plugin leaf classes: TestPlugin
16-11-01 09:47:18.831 [INFO] [acd_cli] - TestPlugin attached.
16-11-01 09:47:18.831 [INFO] [acd_cli] - Settings path is "/home/ebyrne/.config/acd_cli".
16-11-01 09:47:18.832 [INFO] [acdcli.api.client] - Initializing ACD with path "/home/ebyrne/.cache/acd_cli".
16-11-01 09:47:18.832 [INFO] [acdcli.api.oauth] - AppspotOAuthHandler initialized
16-11-01 09:47:19.358 [INFO] [acdcli.cache.schema] - DB schema version is 2.
Getting changes16-11-01 09:47:19.359 [INFO] [acdcli.api.metadata] - Getting changes with checkpoint "None".
16-11-01 09:47:19.359 [INFO] [acdcli.api.backoff_req] - POST "https://cdws.us-east-1.amazonaws.com/drive/v1/changes"
16-11-01 09:47:19.361 [INFO] [requests.packages.urllib3.connectionpool] - Starting new HTTPS connection (1): cdws.us-east-1.amazonaws.com
...
16-11-01 09:47:22.720 [INFO] [acdcli.api.metadata] - Found "reset" tag in changes.
16-11-01 09:47:23.116 [INFO] [acdcli.cache.schema] - Dropped all tables.
16-11-01 09:47:23.641 [INFO] [acdcli.cache.schema] - DB schema version is 2.
Inserting nodes16-11-01 09:47:23.715 [INFO] [acdcli.cache.sync] - Inserted/updated 2 folder(s).
16-11-01 09:47:24.332 [INFO] [acdcli.cache.sync] - Inserted/updated 1998 file(s).
16-11-01 09:47:24.449 [INFO] [acdcli.cache.sync] - Parented 2000 node(s).
.16-11-01 09:47:24.932 [INFO] [acdcli.cache.sync] - Inserted/updated 40 folder(s).
16-11-01 09:47:25.123 [INFO] [acdcli.cache.sync] - Inserted/updated 113 file(s)
16-11-01 09:47:25.206 [INFO] [acdcli.cache.sync] - Parented 153 node(s).
.16-11-01 09:47:25.356 [INFO] [acdcli.api.metadata] - 2 page(s) in changes.
$ acdcli -v mount /home/sketch/acd/
16-11-01 09:48:01.716 [INFO] [acd_cli] - Plugin leaf classes: TestPlugin
16-11-01 09:48:01.716 [INFO] [acd_cli] - TestPlugin attached.
16-11-01 09:48:01.716 [INFO] [acd_cli] - Settings path is "/home/sketch/.config/acd_cli".
16-11-01 09:48:01.718 [INFO] [acdcli.api.client] - Initializing ACD with path "/home/sketch/.cache/acd_cli".
16-11-01 09:48:01.718 [INFO] [acdcli.api.oauth] - AppspotOAuthHandler initialized
16-11-01 09:48:01.727 [INFO] [acdcli.cache.schema] - DB schema version is 2.
16-11-01 09:48:01.727 [CRITICAL] [acd_cli] - Root node not found. Please sync.
ebyrne242 commented 8 years ago

Interestingly, if I copy ~/.cache/acd_cli from a working machine, mounting works fine. Hopefully having the same auth data on two machines doesn't cause any issues...

galenjdove commented 8 years ago

I am having the exact same issue. acd_cli running fine on 3x machines for the past few months. Now on one of the boxes (running ubuntu 16.04) I am getting the root node not found error. I did a full sync, clear out all cached data, reinstall acd_cli with no resolution. I copied the entire .cache/acd_cli/* folder from a working server and it worked fine for a few hours, unfortunately it gives the root node not found error eventually.

madknave commented 8 years ago

I have a similar situation, not quite the same, but related I would say. The sync command doesn't seem to be getting all the nodes from ACD, resulting on missing folders/files. This doesn't seem to be happening in a different server that I have.

This also create errors with the upload command saying that certain folders already exist, asking me to sync, but with no luck since I'm unable to get all the nodes.

bgemmill commented 8 years ago

sync appears to have a per host backoff on the amazon end, implemented in terms of number of nodes it'll give out to clients

I had to old-sync for a while until that backoff expired and amazon gave me enough nodes via sync to continue.

The best way to fix this is to old-sync once to get an up-to-date nodes.db, and then use that nodes.db for a week or so. At the end sync will work again.

nbyloff commented 8 years ago

I have this same issue just started today. I had a corrupt DB somehow, and from another issue it said to just delete the contents of .cache/acd_cli. At first I just deleted nodes.db, but that didn't work, so I deleted everything in that directory to be sure.

Now after I redid the auth, and ran acd_cli sync, it still will not mount and I get this error. I ran this query from another issue on the new nodes.db, and it returns 0 results.

SELECT *, created FROM nodes WHERE type IS 'folder' AND name is NULL;

edit also just updated to the most current version on master, ran acdcli delete-everything, then got a new oauth_data file. Running acdcli sync gives the Root node not found error, but had a python error this time too:

user@server:~$ acdcli sync
Getting changes.....
Inserting nodes....
16-11-20 21:50:26.641 [ERROR] [acd_cli] - Root node not found. Sync may have been incomplete.
Traceback (most recent call last):
  File "/usr/local/bin/acdcli", line 9, in <module>
    load_entry_point('acdcli==0.3.2', 'console_scripts', 'acdcli')()
  File "/usr/local/bin/acd_cli.py", line 1679, in main
    ret = args.func(args)
  File "/usr/local/bin/acd_cli.py", line 778, in sync_action
    return ret | ERROR_RETVAL
TypeError: unsupported operand type(s) for |: 'NoneType' and 'int'

Any suggestions for a temp fix?

yadayada commented 8 years ago

Any suggestions for a temp fix?

acdcli psync /

nbyloff commented 8 years ago

Trying to replicate this issue, can't anymore. I had an old copy of nodes.db on the server and updating that worked. If I replicate how this happened, I'll report back.

nbyloff commented 7 years ago

I had to rebuild my machine, and on a brand new install yesterday, I got this error trying to do an initial acdcli sync after the acdcli init.

[ERROR] [acd_cli] - Root node not found. Sync may have been incomplete.

So then I ran acdcli psync / then acdcli sync again, and everything worked OK. But what I have noticed is when I mount the drive, it's not mounting all the folders. In fact, it's missing a lot of data in the mount so I assumed the sync was incomplete.

Moving and backing up the current nodes.db* files and trying to sync again results in the same issue. Here's a result of running acdcli -v sync with nodes.db file deleted from ~/.cache/acd_cli

acdcli -v sync
16-12-19 09:13:46.667 [INFO] [acd_cli] - Plugin leaf classes: TestPlugin
16-12-19 09:13:46.667 [INFO] [acd_cli] - TestPlugin attached.
16-12-19 09:13:46.668 [INFO] [acd_cli] - Settings path is "/home/backup/.config/acd_cli".
16-12-19 09:13:46.668 [INFO] [acd_cli] - Cache path is "/home/backup/.cache/acd_cli".
16-12-19 09:13:46.669 [INFO] [acdcli.api.client] - Initializing ACD with path "/home/backup/.cache/acd_cli".
16-12-19 09:13:46.669 [INFO] [acdcli.api.oauth] - AppspotOAuthHandler initialized
16-12-19 09:13:47.719 [INFO] [acdcli.cache.schema] - DB schema version is 2.
Getting changes16-12-19 09:13:47.720 [INFO] [acdcli.api.metadata] - Getting changes with checkpoint "None".
16-12-19 09:13:47.721 [INFO] [acdcli.api.backoff_req] - POST "https://cdws.us-east-1.amazonaws.com/drive/v1/changes"
......
16-12-19 09:13:53.556 [INFO] [acdcli.api.metadata] - Found "reset" tag in changes.
16-12-19 09:13:53.974 [INFO] [acdcli.cache.schema] - Dropped all tables.
16-12-19 09:13:54.652 [INFO] [acdcli.cache.schema] - DB schema version is 2.
Inserting nodes16-12-19 09:13:54.843 [INFO] [acdcli.cache.sync] - Inserted/updated 137 folder(s).
16-12-19 09:13:55.544 [INFO] [acdcli.cache.sync] - Inserted/updated 1861 file(s).
16-12-19 09:13:55.733 [INFO] [acdcli.cache.sync] - Parented 1998 node(s).
.16-12-19 09:13:55.970 [INFO] [acdcli.api.metadata] - Found "reset" tag in changes.
16-12-19 09:13:56.507 [INFO] [acdcli.cache.schema] - Dropped all tables.
16-12-19 09:13:57.363 [INFO] [acdcli.cache.schema] - DB schema version is 2.
16-12-19 09:13:57.473 [INFO] [acdcli.cache.sync] - Inserted/updated 53 folder(s).
16-12-19 09:13:57.896 [INFO] [acdcli.cache.sync] - Inserted/updated 945 file(s).
16-12-19 09:13:58.017 [INFO] [acdcli.cache.sync] - Parented 998 node(s).
.16-12-19 09:13:58.429 [INFO] [acdcli.cache.sync] - Inserted/updated 126 folder(s).
16-12-19 09:13:59.131 [INFO] [acdcli.cache.sync] - Inserted/updated 1874 file(s).
16-12-19 09:13:59.298 [INFO] [acdcli.cache.sync] - Parented 2000 node(s).
.16-12-19 09:13:59.778 [INFO] [acdcli.cache.sync] - Inserted/updated 291 folder(s).
16-12-19 09:14:00.467 [INFO] [acdcli.cache.sync] - Inserted/updated 1709 file(s).
16-12-19 09:14:00.623 [INFO] [acdcli.cache.sync] - Parented 2000 node(s).
.16-12-19 09:14:01.004 [INFO] [acdcli.cache.sync] - Inserted/updated 50 folder(s).
16-12-19 09:14:01.316 [INFO] [acdcli.cache.sync] - Inserted/updated 407 file(s).
16-12-19 09:14:01.494 [INFO] [acdcli.cache.sync] - Parented 457 node(s).
.16-12-19 09:14:01.685 [INFO] [acdcli.api.metadata] - 5 page(s) in changes.

16-12-19 09:14:01.690 [ERROR] [acd_cli] - Root node not found. Sync may have been incomplete.

It might be relevant, but there's thousands of files in these dirs. I noticed two nodes seem to be at a flat 2000. There isn't 2000 files in any given folder. I think the max files in one folder is 900, but most have only a few.

Like I picked one folder, and did a partial sync on it, and now when I mount, it shows the folders that were missing, but the contents of those folders are still empty when they're not on Amazon. So I would have to go through and do a psync on all folders to get caught up?

Any way to bump the 2000 max that I may be seeing (if that's what it is), or how to recursively sync all folders from scratch?

EDIT: looking in acd_cli.py, I see psync has a recursive option. Trying that now to see if it will help catch me up

nbyloff commented 7 years ago

I was able to resolve my issue by using acdcli psync -r <folder path> for different folders that contained more than 2000 files in them.

Looking at other people's issue in here, I think that might be the cause; can anyone else confirm? This issue might be resolved with error messaging that suggests the user try psync or even add the recursive option to the sync command.

Something like :

Root node not found. Sync may have been incomplete.
Do you have more than 2000 files? Try the -r recursive option.
NOTE: Depending on your ACD size, this might take a while.
erse80 commented 7 years ago

hi i have similar error

with command acdcli psync -r i have sync folder with less 2000 files (all work)

but i have 1 folder with 35.000 files and sync it's not possibile. I saw that if I do different psync i receive different error

acdcli -v psync -r ...... 16-12-21 22:46:19.219 [ERROR] [acdcli.api.backoff_req] - Error getting node list. 16-12-21 22:46:19.231 [ERROR] [acd_cli] - Sync failed: RequestError: 500, {"logref":"e800149b-c7c6-11e6-b470-9bcffee82adc","message":"Internal Server Error"}

acdcli -v psync -r ...... 16-12-21 22:49:42.602 [ERROR] [acdcli.api.backoff_req] - Error getting node list. 16-12-21 22:49:42.615 [ERROR] [acd_cli] - Sync failed: RequestError: 429, {"logref":"6121e81d-c7c7-11e6-86df-a50837f2f870","message":"System unavailable. Please back off for a while and try again later"}

any suggestions?

erse80 commented 7 years ago

I think in the end this is the problem for big folder

in debug mode

acdcli -d psync -r /Folder .... reply: 'HTTP/1.1 429 Too Many Requests\r\n'

idgf commented 7 years ago

I am having the same issue. What folder path should I be using for the acdcli -v psync -r < folder path >? I have tried the path on the mount and the path from cloudrive URL but nothing is being resolved. Thanks.

Edit: I took a look at the acd_cli logs and was able to determine the file path to use. Thanks.

ghost commented 7 years ago

I think in the end this is the problem for big folder

Could be a different issue, but I got Root node not found. Please sync. on a drive with 4 dirs and 2 files. Error started right after acdcli init, making a sync resulted in Root node not found. Sync may have been incomplete..

acdcli -d psync / solved the error and fixed the normal sync too. Nothing higher than INFO in the log.

The db was fine though, the remote tree was complete even before psync.

Running acd_cli in a python 3.5 virtualenv in Debian testing. pip3 from Debian repos.

alstonhuang commented 7 years ago

I tried to do acd_cli sync but fail and get the message below: Traceback (most recent call last): File "/usr/local/bin/acd_cli", line 9, in load_entry_point('acdcli==0.3.2', 'console_scripts', 'acd_cli')() File "/usr/local/bin/acd_cli.py", line 1652, in main cache = db.NodeCache(CACHE_PATH, SETTINGS_PATH, args.check) File "/usr/local/lib/python3.4/dist-packages/acdcli/cache/db.py", line 73, in init raise IntegrityError('Could not uniquely identify root node.') acdcli.cache.db.IntegrityError: 'Could not uniquely identify root node.'

I have no idea about what to do next. Can any one help? Thanks.