Import ends incomplete

[axelweichert]

My OneDrive Size is about 866 GB, which i want to migrate to my managed NextCloud installation. First it starts importing my files, creating all follders in the dedictaed subdirectory. But after a time, where about 6 - 10 GB has been imported, the import ends and the notification tells me, that 1 files has been imported.

Now, when i restart the import process, it first tells me, that the import process will start soon und then starts the process.

Is there any know issue or anything i can do?

Kind regards,

Axel

[axelweichert]

I just have look to my Logs:

"message":"OneDrive API error : Client error: GET https://graph.microsoft.com/v1.0/me/contacts resulted in a 429 Too Many Requests

Error Login in the App:

=================================================== "time":"2021-02-11T07:26:14+00:00","remoteAddr":"","user":"--","app":"integration_onedrive","method":"","url":"--","message":"OneDrive API error : Client error: GET https://graph.microsoft.com/v1.0/me/drive/root://Instagram:/children resulted in a 400 Bad Request "time":"2021-02-11T07:26:14+00:00","remoteAddr":"","user":"--","app":"integration_onedrive","method":"","url":"--","message":"OneDrive API error : Client error: GET https://graph.microsoft.com/v1.0/me/drive/root://Kunden:/children resulted in a 400 Bad Request

Failed to get number of contacts: "Client error: GET https:\/\/graph.microsoft.com\/v1.0\/me\/contactfolders resulted in a 429 Too Many Requests response:\n{\r\n \"error\": {\r\n \"code\": \"ApplicationThrottled\",\r\n \"message\": \"Application is over its Request limit.\",\r\n \"inn (truncated...)\n"

What can i do?

[julien-nc]

Thanks for the issue.

Error Login in the App:

What do you mean?

About the 400 Bad Request error, I removed the extra slash that was in the request URL. Let's see if it changes something on your side.

About the 429 Too Many Requests, the number of request is now a lot reduced. Do you have many contact directories in your microsoft account?

So, v0.0.9 is out. Could you try it and tell me if things are better?

[axelweichert]

Now, after the update to 0.0.8 (0.0.9 was not available in the AppStore) it shows teh following: 184 files imported (0 B) (0%) Onedrive import process will begin soon

But nothing starts importing. Before that, an amount of about 100 GB has been imported successfully with some restarts of the migration process. Now, nothing seems to go further than

184 files imported (0 B) (0%) Onedrive import process will begin soon

I have neither stored an contacts or calendars in my private OneDrive.

[julien-nc]

v0.0.9 will appear soon (if it's not already there).

Could you check (just in case) that you don't have any contacts/calendars in your microsoft account?

Even if you didn't launch any contact import, the simple fact to count them could have been the reason for reaching the API request number limit with every releases before v0.0.9.

If you really don't have any this means the API request limit is reached while importing files. Could you check what pops in nextcloud.log when you import the drive?

I guess your server uses the "cron" method for Nextcloud background jobs. Right? Keep in mind the drive import process will start next time Nextcloud's cron task is executed. If you want to speed things up and launch the import directly, you can execute the cron.php script. Assuming that you're using bash on your server, that your webserver user is www-data and that your Nextcloud is installed in /var/www/nextcloud:

sudo -u www-data php -f /var/www/nextcloud/cron.php

Another important thing to know: the import process stops after having downloaded at least 500 MB and schedules another job. The following job is launched when the next cron task is executed.

Btw, do you have enough free space in your Nextcloud to get ~1 TB?

[axelweichert]

Version 0.0.9 is not there at the moment. I will check out later if it is available.

I have only some holidays in there, but no contacts. I never used them.

Then with 0.0.0.9 it should eleminate these 429 requests, right?

My server uses the cron method with schuled cronjobs every 5 min.

Actual nextcloud log output.

----##----##----##----##----##----##----##----##----##----##----

Feb 12 17:08:34 storage nextcloud-nx**[994504]: {"reqId":"bFKMJc03k2WWvcwmewJy","level":2,"time":"2021-02-12T16:08:34+00:00","remoteAddr":"93.227.90.71","user":"a*.****t","app":"integration_onedrive","method":"GET","url":"/apps/in tegration_onedrive/contact-number","message":"OneDrive API error : Client error: GET https://graph.microsoft.com/v1.0/me/contacts resulted in a 429 Too Many Requests response:\n{\r\n \"error\": {\r\n \"code\": \"ApplicationThrottl ed\",\r\n \"message\": \"Application is over its Request limit.\",\r\n \"inn (truncated...)\n","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36","versio n":"20.0.4.0"}

----##----##----##----##----##----##----##----##----##----##----

Client error: GET https://graph.microsoft.com/v1.0/me/contacts resulted in a 429 Too Many Requests

Application is over its Request limit.

I have moved to a managed Nextcloud Installation on Hetzner StorageShare with a limited admin access - a pitty, that i did not know before. But i have about 10 TB of space there, so evough fpr importing 866 GB.

I did not know that the import process is separated in 500 MB jobs and it depends on the scheduled cron jobs.

But, great work you have done, creating this app.

Edit: Just upgraded to 0.0.9. I will write back shortly

[julien-nc]

Well I think the drive import is the cause of reaching the limit and getting 429 response codes. I have to rewrite this in a way that produces less requests. For the moment, the entire directory structure exploration is done on each job instead of catching up where last job stopped. Already imported files are ignored but the exploration itself generates a lot of requests...

So v0.0.9 will most likely not be much better in your case. Now that, thanks to your feedback, I'm more aware of what happens, I can start improving the drive import process.

Thanks for your kind words.

[axelweichert]

Ok that seems to be the reason. Because i hev stored there many - and i mean really many - photos and documents in there, which i wanted to move from OneDrive to my NextCloud. When the initial process of sturcture exploration starts i makes sense, that there a to many requests.

When you something new to test for me, it would be nice if you get in touch with me.

You are welcome.

[julien-nc]

One last question (for today :grin:). When you get a 429 response, does the import really stop (the "import onedrive files" button becomes visible again in user settings)? Or is is just freezing (progress is displayed and does not move)?

I will probably solve this mess by saving a list of known unfinished directories at the end of the import job so that the next job starts from there and only explore those directories.

To be continued. And yes I'll ping you when there is something new to try. Thanks.

[axelweichert]

With 0.0.7 and some manual restarts of the process, which ended with the message "1 file imported". Until this i have imported about 100 GB of data form my OneDrive and the whole diretory structure has been created.

Now with 0.0.9 and with 0.0.7 the process freezes with the message:

184 files imported (0 B) (0%) Onedrive import process will begin soon

Thanks a lot. Happy to hear from you back soon,.

[julien-nc]

Hey, I published a nightly build (v0.0.10-1-nightly) in which each job resumes the import where the previous one stopped. Could you try it?

If you're not comfortable with nightly builds, let's wait for next release.

[axelweichert]

Sure i can try, but i can only see the normal - not-nightly releases, because the hoster of my managed NextCloud allows only stable releases. Any option to publish it as stable? At the moment i can only see 0.0.9

[julien-nc]

Yeah nightly builds must be installed manually or with the --allow-unstable option of occ app:update command. If you have access to your server system, here is an example (to adapt to your context):

sudo -u www-data php occ app:update --allow-unstable integration_onedrive

I would prefer that we both test this nightly before publishing it.

[axelweichert]

Sadly i don't have fill access to the server - too bad that i did not knew before i moved my old installation to the new provider. I think that we maybe have to wait until you release the stable one.

But i wil ask my provider, if they can install version for me.

[julien-nc]

Hey. I published v0.0.10 anyway.

[axelweichert]

Cool, i will try as it will become visible in the appstore. Thanks man. Will come back shortly with a feedback.

[axelweichert]

Seems to be no change. Process still freezes with same message: 184 files imported (0 B) (0%) Onedrive import process will begin soon

I have also disconnected from OneDrive, deleted to the App in the Azure und created a new one, and i have cleared my path where to import the files from OneDrive to NexxtCloud.

Calendar Impiort is working by the way.

[julien-nc]

How long did you wait after having launched the import?

[axelweichert]

For about 4 hours. Should be enough i think. Message is the same, even after i complete recreate the app in the Azure thingy.

[ctml91]

Having a similar issue except it's spewing out 404 errors.

{"reqId":"my639JwFGe6gZ7DD2bX0","level":2,"time":"2022-03-05T07:25:20+00:00","remoteAddr":"","user":"admin","app":"integration_onedrive","method":"","url":"--","message":"OneDrive API error : Client error: `GET https://graph.microsoft.com/v1.0/me/drive/root:%2FMobile+uploads:/children?filter=file+ne+null` resulted in a `404 Not Found` response:\n{\"error\":{\"code\":\"itemNotFound\",\"message\":\"Item does not exist\",\"innerError\":{\"date\":\"2022-03-05T07:25:20\",\"request-id\": (truncated...)\n","userAgent":"--","version":"23.0.0.10","id":"6223a0ba5fb96"}

Size of OneDrive data totals 33 GB but only 2.9 GB was imported.

[herrherrmann]

I’m running into similar issues, it seems. I’m trying to migrate about 1 TB of files from OneDrive, but the app seems to stop importing after ~8.8 GB without any obvious reason or error message. I also cannot find any error logs or similar – would those be in the NextCloud logs (Settings / Administration / Logging) or elsewhere? I’d be very happy to assist with debugging this to get this eventually fixed!

And for what it’s worth, I’ve had similar issues with another cloud provider (pCloud) in the past, where I wanted to migrate my OneDrive files and all files with certain special characters in the file name (e.g. + and ,) were skipped. I wonder if something similar could be afoot here? 🤔

[maexbower]

I run into as well a similiar Problem. It stops for me already after a couple files around 1 GB. I see always this message in the Log (File Names and Identifiers replaced by #). I know from my work with M365 in a professional context that indeed the platform has some strong rate limits when importing or exporting data. But not sure where exactly we are struggeling here. How can I debug the addon more to support you?

[integration_onedrive] Warnung: OneDrive error downloading file #######.jpg : 
Server error: `GET https://public.db.files.1drv.com/#################` resulted in a `503 Service Unavailable` 
response:
<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd' (truncated...)

[SunboX]

I’m running into similar issues, it seems. I’m trying to migrate about 1 TB of files from OneDrive, but the app seems to stop importing after ~8.8 GB without any obvious reason or error message. I also cannot find any error logs or similar – would those be in the NextCloud logs (Settings / Administration / Logging) or elsewhere? I’d be very happy to assist with debugging this to get this eventually fixed!

And for what it’s worth, I’ve had similar issues with another cloud provider (pCloud) in the past, where I wanted to migrate my OneDrive files and all files with certain special characters in the file name (e.g. + and ,) were skipped. I wonder if something similar could be afoot here? 🤔

Same for me. :( Did you manage to get it to import all files?