search

abraunegg / onedrive

OneDrive Client for Linux
https://abraunegg.github.io
GNU General Public License v3.0
10.06k stars 859 forks source link

Single directory download only #519

Closed tigerjack closed 5 years ago

tigerjack commented 5 years ago

As per the title, is there any way to combine --download-only and --single-directory options? At the moment, I've found no way to achieve the objective. I have just tried to create an empty local directory and issue a synchronize, but I'm still stuck in the Processing changes phase (there are a lot of files in the cloud).

The main reason for this is that I have a lot of files that, at the moment, are only online. I would like, however, to download a specific directory and synchronize it using onedrive.

abraunegg commented 5 years ago

@tigerjack As per issue submission guidelines, please update your post with the following:

Bug Report Details

Describe the bug A clear and concise description of what the bug is.

Application and Operating System Details:

abraunegg commented 5 years ago

@tigerjack Just to also answer the direct question - this is possible: OneDrive Folder Structure:

├── 1
├── 2
│   └── 3
│       └── 4
│           └── 5
├── aasdfasdfasdfasfd.txt
├── newfile.txt
└── todo.txt

Syncing top level directory '2' only:

[alex@centos7full onedrive]$ ./onedrive --confdir '~/.config/onedrive-personal' --synchronize --download-only --single-directory '2' --verbose
Using Config Dir: /home/alex/.config/onedrive-personal
Initializing the OneDrive API ...
Opening the item database ...
All operations will be performed in: /home/alex/OneDrivePersonal
Initializing the Synchronization Engine ...
Account Type: personal
Default Drive ID: <redacted>
Default Root ID: <redacted>
Remaining Free Space: 5368708870
Fetching details for OneDrive Root
OneDrive Root exists in the database
Syncing changes from this selected path: 2
Syncing changes from selected OneDrive path ...
Getting path details from OneDrive ...
Applying changes of Path ID: <redacted>
[alex@centos7full onedrive]$
tigerjack commented 5 years ago

@abraunegg Two points:

  1. If the directory is not stored locally, but only in the cloud, the command will return ERROR: The requested local directory does not exist. Please check ~/OneDrive/ for requested path. Apart from the ~/OneDrive/ part (which I assume can be easily modified), the problem is that I have to create the empty local directory first and then I can issue the command. Or at least, I suppose so, since I'm still stuck in syncing, which leads to the other point.
  2. If I just want to download one single directory, I have to perform a full synchronization. Could it be avoided?
abraunegg commented 5 years ago

@tigerjack

the problem is that I have to create the empty local directory first and then I can issue the command.

Correct. It assumes that the local directory already exists and you need to make a local empty directory.

If I just want to download one single directory, I have to perform a full synchronization. Could it be avoided?

Yes. Create the required top level local directory and the client will download only that single directory.

tigerjack commented 5 years ago

@abraunegg Unfortunately, this doesn't seem to be the case.

>onedrive --synchronize  --download-only --single-directory Videos
Initializing the Synchronization Engine ...
Syncing changes from selected OneDrive path ...
Processing 201 changes
Processing 252 changes
Processing 246 changes
....

I also made tests with a cloud directory containing just one single file, but still it tries to resync everything.

abraunegg commented 5 years ago

@tigerjack What you are seeing is changes / items delivered from OneDrive and the client working out what to do with that 'change' - hence 'Processing'

If they 'match', you will see the change / download occur. I am not seeing anything here other than normal operation. Increase verbosity your self (--verbose or --verbose --verbose) to see what is actually going on.

Edit: Also:

onedrive --synchronize --download-only --single-directory Videos

You need to use quotes around your directory as per readme to ensure correct operation:

onedrive --synchronize  --download-only --single-directory 'Videos'

Not that this will 'change' your situation however - but increasing verbosity will show you what your client is actually doing.

tigerjack commented 5 years ago

@abraunegg Ok, increasing the verbosity level gives me a lot of Remote change discarded - not in --single-directory scope. What the problem is, at least in my case, is that I have a lot of online files and the whole synchronization process takes way too long to just download a single directory: it has to download and analyze the whole list of changes before starting to download my single directory. I had to power off my laptop after 10 hours of Processing changes yesterday, but still the process was not completed. Why not just download/analyze the changes of that single directory? Is it a limitation of OneDrive API?

As per your edit: you're right, I was just typing a quick example.

abraunegg commented 5 years ago

@tigerjack It is a limitation of the API - you cannot query just a single directory for changes - you receive all changes from OneDrive which need to be analysed and applied if they match.

tigerjack commented 5 years ago

@abraunegg Ok, it makes sense. However, coming back to the original question, you can see now why the workaround (create a local empty top-level directory and issue a --synchronize --download-only) is extremely time-consuming: it is unnecessary to have the whole list of changes, wait for the whole changes analysis and only then start the download (or synchronization) since the directory we are trying to synchronize is empty (no conflicts at all). Edit: I mean, in case the directory doesn't exist locally, you have to just download it (maybe asking the user for confirmation), without going into the whole process.

abraunegg commented 5 years ago

@tigerjack

Ok, it makes sense. However, coming back to the original question, you can see now why the workaround (create a local empty top-level directory and issue a --synchronize --download-only) is extremely time-consuming: it is unnecessary to have the whole list of changes, wait for the whole changes analysis and only then start the download (or synchronization) since the directory we are trying to synchronize is empty (no conflicts at all).

Actually - if using --single-directory and the local directory does not exist, it will not sync until you create that directory. This is by design, because it is 'assumed' you know what your doing and you know the impacts (local db cache of changes) if you get this wrong.

I see no issue here in the client not downloading / syncing if the directory does not exist locally (even as an empty directory). If it doesn't exist - all you have to do is ensure you do a mkdir <path> first as required.

tigerjack commented 5 years ago

@abraunegg I think what I was really suggesting is that, if the local directory does not exist locally, it may be an option to ask the user if he/she wants to only download the remote directory. In case the answer is positive, it is useless to download the whole list of changes of all the directories and files, analyze it and at the end starts the download: it would suffice to just download the remote directory.

abraunegg commented 5 years ago

@tigerjack I understand what your saying - however:

In case the answer is positive, it is useless to download the whole list of changes of all the directories and files, analyze it and at the end starts the download: it would suffice to just download the remote directory.

This indicates to me your not clear on how the client operates.

Regardless of if the directory to sync exists locally or does not (or a prompt is done to create, or it is created automatically so that it now exists prior to sync) - the OneDrive API will send you 100% of all changes / files from your OneDrive account - not just the directory you want to sync. The client has to iterate through these 1 by 1, and if the 'change' matches, it is processed and the file is downloaded. There is zero way to control this.

The client and the API works in this way, so that today your syncing only the directory called 'foo', but tomorrow you want to sync 'bar' - both 'foo & 'bar' exist on your OneDrive account as root folders, and you want to independently sync these 'when you initiate the sync'. Unless you use 2 separate sync directories & associated configuration files, everything will be stored in the local database tracking 'foo' and 'bar' elements, so that you can sync, at any time 'foo' and then 'bar' separately. You then may choose to sync your entire OneDrive - which again, existing directories would have to be handled appropriately which is what would happen.

tigerjack commented 5 years ago

@abraunegg Ok, now it is more clear. Basically, there is no way to even discharge the list of changes returned by the API call (because there is some kind of unique id involved for each directory?). It's a pity, and I think I must cope with it. Just to double-check on the second part: if I completely endure the synchronization process of the onedrive --synchronize --download-only --single-directory "Videos" today, and tomorrow I issue the onedrive --synchronize --download-only --single-directory "Pictures" command (assuming no changes on remote), I don't have to repeat the whole "Analyzing changes" process? At this point, the download will be immediate, am I right?

abraunegg commented 5 years ago

@tigerjack No - The initial sync is tracking what is 'in-sync' to avoid over aggressive 'Analyzing changes'.

Subsequent changes / syncing 'Pictures' tomorrow - OneDrive will ONLY send you the delta links as the client inform's OneDrive what is 'in-sync' and what is needed. 2nd and future sync's are MUCH quicker - it is only the initial 'get me in sync' that is slow and unavoidable.

If you however issue a --resync command - then your local state is removed and ..... well .... all OneDrive changes will be sent again.

tigerjack commented 5 years ago

@abraunegg Ok, now everything is more clear. Thanks for your help (and for all the work on this project).

abraunegg commented 5 years ago

@tigerjack Also - noticed your version your running is 'very old' - ensure that you are running the latest 'master' release to ensure your not tripped up by fixed bugs / issues.

tigerjack commented 5 years ago

@abraunegg mmm, I compiled it yesterday, maybe something went wrong during the sudo make install. I'll double check now. Thanks for the tip.

tigerjack commented 5 years ago

@abraunegg I don't want to derail this issue, but it seems that I had the right version installed. However, from the makefile, I can see that the version is echoed to a version file and taken from the git describe --tags: if those are not explicitly fetched from remote, the output of onedrive --version is wrong.

EDIT: Additionally, https://github.com/abraunegg/onedrive/blob/master/Makefile.in#L143 always returns false for me, even if the file exists. I don't know if it is expected.

abraunegg commented 5 years ago

@tigerjack Can you list out how you are cloning & compiling?

The usual process is (for example):

git clone https://github.com/abraunegg/onedrive.git
source ~/dlang/dmd-2.085.0/activate
cd onedrive
./configure
make clean; make;
deactivate
sudo make install

Make output (for current master) should be:

if [ -f .git/HEAD ] ; then \
        echo v2.3.3-17-gef6af8e > version ; \
else \
        echo 2.3.4dev > version ; \
fi
dmd  -w -g -O -J. -L-lcurl -L-lsqlite3  -L-ldl src/config.d src/itemdb.d src/log.d src/main.d src/monitor.d src/onedrive.d src/qxor.d src/selective.d src/sqlite.d src/sync.d src/upload.d src/util.d src/progress.d -ofonedrive
tigerjack commented 5 years ago

@abraunegg I cloned the repository long ago and maybe I didn't force the tags download. Indeed, the git describe --tags returned an empty string and I had to manually execute git fetch --tags. A few days ago I just pulled and recompiled. Also, I'm on Arch and dmd is installed as a package, so I don't think I should activate/deactivate it directly, am I right?

abraunegg commented 5 years ago

@tigerjack If you delete your 'onedrive' directory & reclone - it 'should' work correctly however.

Also, I'm on Arch and dmd is installed as a package, so I don't think I should activate/deactivate it directly, am I right?

If using a package - yes - however sometimes the package version of DMD/LDC is also way out of date for some distro's.

lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.