rjonkman
commented
3 days ago Can I just delete the onedrive config folder as well and my OneDrive folder and start from scratch?
Open rjonkman opened 3 days ago
rjonkman
commented
3 days ago Can I just delete the onedrive config folder as well and my OneDrive folder and start from scratch?
abraunegg
commented
3 days ago There are so many bugs, I can't even begin. Don't update
Explain this in more detail please.
Can I just delete the onedrive config folder as well and my OneDrive folder and start from scratch?
Of course you can .....
rjonkman
commented
3 days ago I'll try that. I'm not going to make any formal bug reports because it's so hard to pin down the issue. The --dry-run flag showed the expected output, but without the --dry-run flag, the client crashed when trying to download.
I also had lots of issues with multiple locations of configuration files. I had two sets on my system, probably because I previously install from apt. Lot's of headaches. I hope I can get it working without having to downgrade back to version 2.4.
abraunegg
commented
3 days ago I'll try that. I'm not going to make any formal bug reports because it's so hard to pin down the issue. The --dry-run flag showed the expected output, but without the --dry-run flag, the client crashed when trying to download.
Please list out all your issues .....
The client should not crash - unless you were using v2.5.0 initially as it had a few issues that were fixed, which unfortunatly caused an unknown other issue v2.5.1 (major crash due to timestamps) which was fixed with v2.5.2.
The entire v2.5.x is a 100% rewrite.
It went through 5 x alpha releases + 3 RC releases over a 9 month period.
350+ use cases were extensivly tested and validated.
There is change in functionality - that is not a bug
rjonkman
commented
3 days ago So I started from scratch. The config file only contains:
sync_dir = "/home/rjonkman/OneDrive"
And using a 'sync_list' file which contains:
/coding
/MIDI Expression
/MIDI Expression DrumRunning OneDrive from the terminal with:
onedrive --sync --resync --dry-run --verbose
Ends with the following output:
Including file - included by sync_list config: coding/MEC/source/Routing.h
Parental Path structure needs to be created to support included file: coding/MEC/source100% normal output
rjonkman
commented
3 days ago So crashing 100% of the time is normal output? My OneDrive folder is empty. It should be mock downloading files.
rjonkman
commented
3 days ago And here is the additional output with --verbose -v+
DEBUG: Parental Path structure needs to be created to support included file: coding/MEC/source
DEBUG: CurlEngine getCurlInstance() called
DEBUG: CurlEngine curlEnginePool current size: 3
DEBUG: CurlEngine was in a valid state - returning existing CurlEngine instance
DEBUG: CurlEngine instance ID: Z1htVBJ2srT8cgSq
DEBUG: Read token from appConfig
DEBUG: Authorised State: true
DEBUG: createLocalPathStructure input onedriveJSONItem: {"cTag":"aYzpDQUU3MjA3REFDMEYxMDAxITIyMDI1Ny40MjQ","eTag":"aQ0FFNzIwN0RBQzBGMTAwMSEyMjAyNTcuMTcw","file":{"hashes":{"quickXorHash":"YyDmGyxRmEqsmir7r0iZRqFNMNQ=","sha1Hash":"85E16D658E2465EB0E62E2C04CFFBAB240597438","sha256Hash":"503B2EB4D2209D9F4B43710F38CD509F4A527AC3E7F0BEC13690A32031A28375"},"mimeType":"text\/plain"},"fileSystemInfo":{"createdDateTime":"2024-07-06T22:07:18.356Z","lastModifiedDateTime":"2024-09-04T04:42:26Z"},"id":"CAE7207DAC0F1001!220257","name":"Routing.h","parentReference":{"driveId":"cae7207dac0f1001","driveType":"personal","id":"CAE7207DAC0F1001!44416","name":"source","path":"\/drive\/root:\/coding\/MEC\/source"},"size":23422}
DEBUG: Request URL = https://graph.microsoft.com/v1.0/drives/cae7207dac0f1001/items/CAE7207DAC0F1001!44416?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size,webUrl,lastModifiedBy,lastModifiedDateTime
DEBUG: Existing Microsoft OneDrive Access Token Expires: 2024-Oct-02 14:28:09.7369291There is no evidence of a crash .... you are also doing a dry run .. so application output is slightly different.
If this is crashing .. whats the full output on your screen including any crash debug lines
abraunegg
commented
3 days ago And here is the additional output with --verbose -v+
DEBUG: Parental Path structure needs to be created to support included file: coding/MEC/source DEBUG: CurlEngine getCurlInstance() called DEBUG: CurlEngine curlEnginePool current size: 3 DEBUG: CurlEngine was in a valid state - returning existing CurlEngine instance DEBUG: CurlEngine instance ID: Z1htVBJ2srT8cgSq DEBUG: Read token from appConfig DEBUG: Authorised State: true DEBUG: createLocalPathStructure input onedriveJSONItem: {"cTag":"aYzpDQUU3MjA3REFDMEYxMDAxITIyMDI1Ny40MjQ","eTag":"aQ0FFNzIwN0RBQzBGMTAwMSEyMjAyNTcuMTcw","file":{"hashes":{"quickXorHash":"YyDmGyxRmEqsmir7r0iZRqFNMNQ=","sha1Hash":"85E16D658E2465EB0E62E2C04CFFBAB240597438","sha256Hash":"503B2EB4D2209D9F4B43710F38CD509F4A527AC3E7F0BEC13690A32031A28375"},"mimeType":"text\/plain"},"fileSystemInfo":{"createdDateTime":"2024-07-06T22:07:18.356Z","lastModifiedDateTime":"2024-09-04T04:42:26Z"},"id":"CAE7207DAC0F1001!220257","name":"Routing.h","parentReference":{"driveId":"cae7207dac0f1001","driveType":"personal","id":"CAE7207DAC0F1001!44416","name":"source","path":"\/drive\/root:\/coding\/MEC\/source"},"size":23422} DEBUG: Request URL = https://graph.microsoft.com/v1.0/drives/cae7207dac0f1001/items/CAE7207DAC0F1001!44416?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size,webUrl,lastModifiedBy,lastModifiedDateTime DEBUG: Existing Microsoft OneDrive Access Token Expires: 2024-Oct-02 14:28:09.7369291
No crash .....
rjonkman
commented
3 days ago Then while isn't it showing the list of files that it's going to 'pretend' to download. It certainly did when I didn't have a sync_list file.
abraunegg
commented
3 days ago Additionally - your curl version is this:
7.81.0Please read this very carefully:
https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#compatibility-with-curl
abraunegg
commented
3 days ago Then while isn't it showing showing the list of files that it's going to 'pretend' to download. It certainly did when I didn't have a sync_list file.
Application output has changed.
The client will first scan everything online first, before doing anything.
rjonkman
commented
3 days ago so if I go and delete my OneDrive folder one more time, the following should download all my files?
onedrive --sync --resync --verbose -v+ --force-http-11
abraunegg
commented
3 days ago Test with --dry-run
Dont do debug mode - just use --verbose once
rjonkman
commented
3 days ago So this?
onedrive --sync --resync --verbose --dry-run --force-http-11
I'm pretty sure we already had this conversation, and it wasn't 'mock-downloading' anything, but I will try again.
abraunegg
commented
3 days ago So this?
onedrive --sync --resync --verbose --dry-run --force-http-11I'm pretty sure we already had this conversation, and it wasn't 'mock-downloading' anything, but I will try again.
I would enable logging to a separate file so you also have a detailed log file of actions
rjonkman
commented
3 days ago Great. How do I do that?
abraunegg
commented
3 days ago Great. How do I do that?
Please read:
https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#enabling-the-client-activity-log
rjonkman
commented
3 days ago For those who like direct answers.
--enable-logging
rjonkman
commented
3 days ago So here we go with:
onedrive --sync --resync --verbose --dry-run --force-http-11 --enable-logging
The log shows exactly the same results as the terminal window. There is no list of files to be downloaded. Is this expected?
abraunegg
commented
3 days ago Can you peovide the full log for:
So 2 full logs via email
rjonkman
commented
3 days ago I can give you the log, but it has sensitive data that I'm going to share publicly. How should we proceed.
abraunegg
commented
3 days ago I can give you the log, but it has sensitive data that I'm going to share publicly. How should we proceed.
Please read this very carefully:
https://github.com/abraunegg/onedrive?tab=readme-ov-file#reporting-an-issue-or-bug
rjonkman
commented
3 days ago Sent
rjonkman
commented
3 days ago Unfortunately after wasting 6 hours on this today, I had to go back to version 2.4. I built it from sources as the Ubuntu apt source seems to be gone. I sure wish it supported the sync_list feature, but at least it works. Hopefully the logs I provided can help track down the issue and I can test again.
abraunegg
commented
3 days ago Unfortunately after wasting 6 hours on this today, I had to go back to version 2.4. I built it from sources as the Ubuntu apt source seems to be gone. I sure wish it supported the sync_list feature, but at least it works. Hopefully the logs I provided can help track down the issue and I can test again.
I do not undestand what you mean by v2.4.x not supporting 'sync_list' .... you can review the v2.4.25 documentation here: https://github.com/abraunegg/onedrive/tree/v2.4.25
rjonkman
commented
3 days ago Well if it does support 'sync_list', then that's good news.
abraunegg
commented
2 days ago I built it from sources as the Ubuntu apt source seems to be gone .. curl version: 7.81.0
Given you are using Ubuntu (your choice .... ) so you are using no doubt a broken curl version (got to love a distribution providing users with broken packages in the guise of stability .. again your choice). To work around these 'issues' add to a 'config' file the following options:
force_http_11 = "true"
ip_protocol_version = "1"These are valid for v2.4.25 and v2.5.2
I am going to close this issue due to:
rjonkman
commented
2 days ago Really what's going on here is that you just don't care enough--as long as it works for you. I understand. You don't get paid and you don't have the proper debugging tools.
abraunegg
commented
2 days ago Really what's going on here is that you just don't care enough--as long as it works for you. I understand.
10000000000000000000000000000% inaccurate statement and total incorrect assumption on your behalf. Please do some reading on all other closed issues|discussions where I have gone way above and beyond to help users with issues, diagnose what is going on and fix issues ........
My config file does contain force_http_11 = "true" and ip_protocol_version = "1"
All your logs and data and information do not reflect this this statement and have been 100% inconsistent in this regard.
You don't get paid and you don't have the proper debugging tools.
Open Source Development is not about getting paid - it is about giving back. Debugging tools and knowledge I have plenty of ... users that have issues that cannot be replicated despite trying need to understand that their assistance in diagnosing the issue has to happen - and there will be a ask of do this/capture that .. then that needs to be analysed to look deeper , and all log files you provided thus far - they do show that the client is exiting for some reason (processing just stops) ... but with 100% zero reason (zero crash evident). I have offered to look deeper into this but you have chosen to revert to an older version and leave it there. Based on all of that it leads me to think that there is some sort of issue with curl itself on your system which reflects itself in v2.4.x where the client just 'stops' with no reason ... your exact issue.
So ... I go back to the following based on historic issues and problems:
I would rather work out what is going wrong ... however information from you has not been full or correct based on the ask.
I 100% suspect this is something unique to your environment and/or how your network connectivity is setup ... but .. that is a major diagnosis that has not been done and you would rather make 100% incorrect statements than actually work through the problem at hand.
rjonkman
commented
2 days ago Oh come on, Alex. I mentioned using the --force-http-11 switch several posts up, and in our person email exchange I mentioned that I updated my config to include force_http_11 = "true" and ip_protocol_version = "1".
I admit that the application did not 'crash', but mysteriously exiting without completing its intended task is essentially the same thing. After providing you multiple logs with every debug option available, you yourself admit that you have no idea why its suddenly exiting, which is the same thing as saying you do not have proper debugging tools to figure what's going on.
Like I said in my post before you closed this issue as 'not a bug', if you have another update for me to test (with something new to help debug), I'll be happy to test it again. Like you, I'm a developer. I have to compile the same code in Windows, macOS, and Linux, and so I need OneDrive working properly while I work.
abraunegg
commented
2 days ago As per email exchange:
echo $? ?rjonkman
commented
1 day ago config
sync_dir = "/home/rjonkman/OneDrive"
force_http_11 = "true"
ip_protocol_version = "1"
threads = "1"command-line
onedrive --sync --resync --verbose --verbose --debug-https > debug_output.log 2>&1
output from 'echo $?'
141
Still nothing downloads. I'll email the log and .sqlite3 file.
rjonkman
commented
1 day ago Shouldn't there be some debugging doing on here? If the file doesn't exist, shouldn't the program exit with some sort of error message? https://github.com/abraunegg/onedrive/blob/3ad139aa252d7e281ee9c438f4c20dbfddde08ba/src/onedrive.d#L792
rjonkman
commented
1 day ago I assume we aren't getting this far, because even if nothing was written into the file, it should still exist right? https://github.com/abraunegg/onedrive/blob/3ad139aa252d7e281ee9c438f4c20dbfddde08ba/src/curlEngine.d#L394
abraunegg
commented
1 day ago Still nothing downloads. I'll email the log and .sqlite3 file.
Something upstream is killing your active connection .... most likely suspects:
Additionally, there are many open bugs with 'curl' from 8.9.x and below regarding SIGPIPE handling.
The method to handle this is to set:
// Explicitly set libcurl options to avoid using signal handlers in a multi-threaded environment
// See: https://curl.se/libcurl/c/CURLOPT_NOSIGNAL.html
// The CURLOPT_NOSIGNAL option is intended for use in multi-threaded programs to ensure that libcurl does not use any signal handling.
// Set CURLOPT_NOSIGNAL to 1 to prevent libcurl from using signal handlers, thus avoiding interference with the application's signal handling which could lead to issues such as unstable behavior or application crashes.
http.handle.set(CurlOption.nosignal,1);So either your version of 'curl' is ignoring what has been set, or, it is yet another 'curl' bug.
Now that 'root cause' for your issue has been potentially identified, I can look at building a platform to attempt to reproduce your issue - and that includes building a whole new network that limits idle network connections + ignores keep alive(s) .. or recompiling the client to forcibly not use them ... I will have to see.
Your DB file is 100% empty, so data ... not surprising.
rjonkman
commented
1 day ago Doesn't look like ufw is the issue. It is inactive on my system.
jonkman@rjonkman-Surface-Pro-3:~$ sudo ufw status
[sudo] password for rjonkman:
Status: inactiveSo something is killing idle TCP connections despite keep alive being set.
Please do a wireshark of your network to identify what is potentially doing this.
rjonkman
commented
1 day ago I'm on a Surface Pro 3, connecting via WIFI to a wireless router. I switch to connecting via by phone's hotspot but it still doesn't work.
rjonkman
commented
1 day ago I think Wireshark is saying that the RST is coming from Microsoft.
21571 141.583795243 40.126.38.102 192.168.149.105 TCP 54 443 → 36106 [RST, ACK] Seq=5400 Ack=2685 Win=0 Len=0
Also from Microsoft Azure:
20566 135.431907281 20.190.141.37 192.168.149.105 TCP 54 443 → 53280 [RST, ACK] Seq=6570 Ack=1918 Win=0 Len=0
rjonkman
commented
1 day ago abraunegg
commented
1 day ago From the provided wireshark, and my analysis:
In your 'wireshark' capture you have lots of network packets out-of-order + duplicates going on. Just as an example:
What Duplicate ACKs Indicate Packet Loss: Duplicate ACKs are sent when the receiving side notices that one or more packets are missing in the sequence. This often triggers retransmissions from the sender. Network Congestion: Congestion can lead to delayed or lost packets, causing the receiver to detect out-of-sequence packets and respond with duplicate ACKs.
Potential Impacts on curl/libcurl: Connection Instability: If the connection between curl and the server is experiencing significant packet loss, it can lead to retransmission delays or connection resets. When curl attempts to reuse or continue a connection after such instability, it may find the connection in an unexpected state, which can lead to a SIGPIPE. Prolonged Idle Periods: Packet loss might result in prolonged idle periods as curl waits for retransmitted packets. This might contribute to connections timing out and being closed due to perceived inactivity
If a server detects numerous duplicate ACKs, which typically indicate packet loss or network congestion, it may choose to reset the connection regardless of the keep-alive setting. This is because excessive duplicate ACKs signal potential instability in the connection. The server might interpret this as a degraded connection state, prompting it to close the connection with a RST (reset) to prevent further resource usage on what it considers an unreliable link. I have no idea how Microsoft interprets duplicate ACK’s or what measures it uses to determine unstable connections.
As per also my original advice regarding your 'curl' version, please attempt to upgrade this in the meantime to something more modern than 7.81.0 , as potentially some of the issue you are seeing is directly related to your use of the old version.
I would look at your WiFi adapter | driver within your Linux kernel to ensure that there are no updates there to be made, or internal firmware for your WiFi adapter.
abraunegg
commented
1 day ago @rjonkman You have the following options:
I have articulated why the differences are there between v2.4.x and v2.5.x. I have also articulated that the way that curl is used, and in v2.4.x it is ‘masking’ the issue you are seeing with v2.5.x – which is why you have not seen it with v2.4.x yet – and you eventually will.
I have 100% zero idea why you will not at least try to upgrade your 'curl' version to resolve this situation for you, but you are happy to upgrade|downgrade the client ..... same action, just a different piece of software.
It is your choice how you proceed here, not mine, however I have given you:
I will keep working on looking to handle the SIGPIPE when I can get free air time - and I am not going to do this with any priority, however for now you have your resolution and it is your choice how you proceed.
Describe the bug
There are so many bugs, I can't even begin. Don't update.
Operating System Details
Client Installation Method
From Source
OneDrive Account Type
Personal
What is your OneDrive Application Version
2.5.2
What is your OneDrive Application Configuration
What is your 'curl' version
Where is your 'sync_dir' located
Local
What are all your system 'mount points'
What are all your local file system partition types
How do you use 'onedrive'
Really?
Steps to reproduce the behaviour
n/a
Complete Verbose Log Output
Screenshots
No response
Other Log Information or Details
No response
Additional context
No response