Rework #2608 with #2661 for testing on physical systems

[abraunegg]

• Rework #2608 with #2661 for testing on physical systems
• Specifically testing these issues from RC1:
  • Re-connect to WiFi and the client will crash with the below error messages for each thread
  • Multiple functions handle errors incorrectly that do not return after retrying the function. As a result, the validation check fails when returning from the recovering retry in the original failed case, even though the operation is actually successful. (#2601)
• General usability against all other 100+ test use cases

[abraunegg]

Test Case 1: Disable WiFi whilst 'Processing API Response Bundle: ...' message occurs

• Start application (either -s or -m .. does not matter)
• When 'Processing API Response Bundle: ...' message occurs, disable WiFi
• Wait for API timeout messages to occur & retry events
• Reconnect WiFi

Expected Outcome:

• Application continues on with fetching API bundle responses and continues application processing

Result:

• Curl debugging is being printed, this needs to be negated and only printed in debug mode or --debug-https mode
  • Fixed via https://github.com/abraunegg/onedrive/pull/2686/commits/ef4e990ef664d5bd269e4873ccbcbf53942447a7
• When WiFi is re-connected, the next 'bundle' is fetched , however the application appears to 'hang' .. given that Internet access has been restored, the client should understand this and continue without issue. The client still thinks that there is a connectivity problem.

Post ef4e990

[abraunegg]

Test Case 2: Flood API with requests to trigger a 429 event

• Start application (either -s or -m .. does not matter)
• Flood API with requests to trigger event
• Determine the application handling

Expected Outcome:

• Only that specific thread that caused / triggered the 429 should be held / paused

Result:

• It appears that all threads are being paused, not the thread that triggered the 429 event

Need to compare this with RC1 and how this is handled

[abraunegg]

Test Case 3: Disable WiFi whilst 'Downloading file: ' is occurring

• Start application (either -s or -m .. does not matter)
• When 'Downloading file: ' message occurs, disable WiFi
• Wait for API timeout messages to occur & retry events
• Reconnect WiFi

Expected Outcome

• Each thread should detail that access has been interrupted
• On re-connect, each thread should download the file and application should continue

Result:

• System acts as expected

[abraunegg]

Test Case 4: Use ^C to cancel application operations whilst 'Waiting for any existing threads to complete'

• Start application (either -s or -m .. does not matter)
• When 'Downloading file: ' message occurs, disable WiFi
• Wait for API timeout messages to occur & retry events
• Reconnect WiFi
• Use ^C to cancel multiple times

Expected Outcome:

• Application exits in the correct manner

Result:

• Application segfaults

Fixed

• Fixed via https://github.com/abraunegg/onedrive/pull/2686/commits/dbe21ed903b985dfc6820b8870dd78e63d185cc8

[abraunegg]

Test Case 5: Deliberately edit 'refresh_token' to make it malformed

• Edit 'refresh_token' and make it malformed by adding just a single new character into the string anywhere

Expected Outcome:

• Application should handle malformed item and advise to perform a --reauth

Result:

• 2608 crashes the application with an unhandled error

• 2661 (RC1) handles this situation correctly

PR2608

PR2661

Application crash due to malformed 'refresh_token' fixed via https://github.com/abraunegg/onedrive/pull/2686/commits/9fa7494d4155f5d1d88c3bd12fbcba5efb25ab01

[abraunegg]

Test Case 6: Run for extended period, deleting | creating new local data randomly in --monitor mode

• Run in --monitor mode
• Run script to generate random data at random intervals

Expected Outcome:

• Memory should remain fairly stable for the process

Result:

• Memory is leaking somewhere, fairly badly ... will need to run another 'valgrind' process against this PR to determine where this is leaking from in comparison to RC1
• Memory usage with RC1 is much better (lower, no noticeable|significant memory leakage due to prior vlagrind work)

Fixed

Fixed via:

• https://github.com/abraunegg/onedrive/commit/2bb82f718fc509ccda3d102a0313671cd2b23430
• https://github.com/abraunegg/onedrive/pull/2686/commits/ec933af290aba82425afaca781b00ecb22048761

[abraunegg]

Test Case 7: Run for extended period, deleting | creating new local data randomly in --monitor mode

• Run in --monitor mode
• Run script to generate random data at random intervals

Expected Outcome:

• Process should not hang / crash and continue operating during the extended run

Result:

• Client stopped processing inotify events / appeared hung after a few hours of operation. Not related to memory leak, plenty of memory (physical and swap left on system). A single CTRL^C and the client exited to the terminal, meaning that something internally crashed, without an error message or anything.
• Need to validate and verify with RC1 as to what occurs there as this was not seen in RC1 extended (>24hrs) constant operational running

[abraunegg]

@bpozdena

If possible - please can you test this PR in your environment in the next few days ?

This PR will become the basis for v2.5.0-rc2

Specifically, looking for you to test and evaluate:

• socket usage / reuse
• curl engine usage / reuse
• anything else socket / curl / dns related which was causing you exhaustion in your environment
• looking for also usage | performance difference between onedrive v2.5.0-rc1-36-g0f012b9 and this PR that you might determine or uncover. For me, with all my soak testing (>24 hrs) running + usage I am not seeing a performance detriment between the two, but would like your feedback.

Notes

• When using --monitor at the end of each 'loop' currently all curl engines are cleared to ensure a clean state for the next loop.
• I would also add display_memory = "true" to your config file so that you can see what memory is being used
• There is currently extra logging output going on regarding pool size etc

[abraunegg]

Closed due to #2709

[abraunegg]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.