Notes deleted from disk locally after updating to new python3 update, 100 notes remain

[casutherland]

Hi @swalladge. Thanks for your efforts on the unicode issue and for keeping sncli development moving!

I will try to dive into debugging this further at some point, or understanding the new python3 code.

Quick question for now: is there a new synchronization and local caching strategy in place with the new python3 code, to only cache 100 notes locally?

Note: I have a large 1500 note database, and use Notational Velocity (nvALT) on other platforms for research notes.

So, earlier this week when I tested the python3 branch on swalladge/sncli, before you merged the python3 commits with insanum/sncli master, I noticed that a large number of notes (actually all notes looking at the log files again) were deleted as soon as the python3 instance was launched. It appeared that only 100 notes were synced and saved to disk, from apparently arbitrary time frames.

I also tried initializing a brand new .sncli local database with the python3 code, with the same result (100 seemingly arbitrary notes are synced and saved to disk).

I tabled the issue and didn't report it on your fork, as I thought it might just be something unique to my environment or your fork, and I saw that you were going to unfork.

However, this morning, I tried again after updating to the new python3 code at the head on insansum/sncli master, and the same behavior was observed.

I'll try to debug further later.

Meanwhile, I have reverted to commit d0e75dd on insansum/sncli with a fresh first-time sync, and all is well in python2 sncli land. : )

[1] FWIW: to the log

2016-05-27 00:30:26,136 [DEBUG] Full sync completed  # pre-python3
2016-05-27 00:32:23,172 [DEBUG] sncli logging initialized # first python3 initialization
2016-05-27 00:32:26,417 [DEBUG] Sync worker: started
2016-05-27 00:32:26,418 [DEBUG] Starting full sync
2016-05-27 00:32:26,480 [INFO] Starting new HTTPS connection (1): simple-note.appspot.com
2016-05-27 00:32:26,859 [DEBUG] Setting read timeout to None
2016-05-27 00:32:26,942 [DEBUG] "POST /api/login HTTP/1.1" 200 64
2016-05-27 00:32:26,966 [INFO] Starting new HTTPS connection (1): simple-note.appspot.com
2016-05-27 00:32:27,242 [DEBUG] Setting read timeout to None
2016-05-27 00:32:27,349 [DEBUG] "GET /api2/index?auth=<>&length=<>&email=<>HTTP/1.1" 200 20896
2016-05-27 00:32:27,459 [DEBUG] Deleted note from disk (key=<>.... #followed by >1500 Deleted note from disk messages

[samuelallan72]

Thanks for picking that up! (I have just under 100 notes by the way, so wouldn't have discovered that!)

As far as I know, I didn't change anything regarding the notes database. It could be that something inadvertently was changed when porting network operations to the requests library though (getting an incorrect value for number of notes for example) - so that would possibly be the first place to check. I'll look into it as soon as possible.

Yes - I'll likely only be using my fork for random testing now - I'll be fixing bugs and such directly here (hopefully).

[samuelallan72]

The above commit should solve the specific problem you are getting. The issue was that I hadn't updated a loop condition to use the response object (so the loop to keep getting blocks of 100 notes never executed).

[casutherland]

Tested. Appears to have resolved the issue. Thanks @swalladge.