Backup to Google Drive

[blackforestboi]

@ShishKabab here are the documents describing the user flow for the backup service.

Workflows

A. General Backup:

1. Backup settings page > also change menu and add submenus
2. Selection of provider, then popup with auth mechanism to relevant provider. in this case google.
3. When provider confirmed, "Start Backup" button appears
4. Progress bar
5. View when token has expired and backup has been paused. In this case we also need to send a notification via the Memex notification system. You just need to prepare everything, I'll do the text. (@mukeshkharita built it, so he may be able to answer how you dynamically can publish notifications, without hardcoding them)
6. Upsell process. I did it all via woocommerce for now, because all payment processing stuff is already implemented there. But would also favor a switch later, in case its doable. Reason is also that people may get in touch with other crowdfunding projects there.
7. checkout page: has signup process in place already. Question here: How do we do the authentification flow in the payment process. As you can see in the checkout form, there is already a way to enter username and password, and AFAIU in the following article it describes that you can hook the account creation directly to an oauth server. So it might not be necessary to re-route. anything.
8. Order confirmation. Here it would be good if we had the chance to re-route back to the settings.
9. When user returns to the settings page, it should show the "upgrade" button now as "enabled". If possible without users even needing to reload.
10. disabling a running subscription.
11. Re-enabling a subscription when it was cancelled but still running.

Mockups:

backup-upgrade.png.zip

Other Questions:

1. When a user upgrades to "automatic backup" while doing the manual backup, what happens then?
2. Do we offer settings for different backup prefs in the beginning, and if not when? Asking if I should already consider that use case in the mockups.
3. when a user does their initial backup, do we let that run in the background? It is limited by the google cookie that does not refresh right? So if we let it expire after 24h a user has practically "automatic" backup for that period? (which I don't find too problematic. I would still find it a PITA to press that button once a day.

[ShishKabab]

About the general flow: 1) I'd change the copy of the Google cookie expiry thing. Too technical. Let's say something like 'The backup took too long, so we paused it. Click here to continue backing up.' Technical note: when the user aborts his session and comes back, the progress bar will reset, because it's treated like a new incremental backup. 2) I'd make the 'soon' options for the providers a bit more action inspiring. Let's make the users fund or vote for these things. 3) The user will get redirected to Google and back again. No pop-up. Simplest to implement for now. 4) WP will need to know where to redirect to after the purchase. Do they have some documented mechanism in Woocomerce which we can use to pass along the URL of the backup page.

The other questions: 1) The backup continues in the bg. When a user comes back, they'll just see more progress, and an indication they now have automatic back-ups. 2) Nope, not yet. Introduces lots of technical complexity. 3) Indeed. However, we don't trigger the real automatic back-up in the extension if not payed for. So the backup is only made until the point the user initiated the backup.

[blackforestboi]

Adding information about the experience bugs here: 1) Still get stuck on the sign-in page Dont get any errors, but that is what is in the console both of the background page and the sign-in page

There is also a cancelled callback in the network tab?

2) There is this warning error, that came up when I switched to another tab. The sidebar is disabled though, so I wonder why this error exists? @poltak

[mukeshkharita]

Hi @ShishKabab There are some observations when I run the tests on backup.

When I log in, stuck out to the login page, get the error.

All things work fine. Great work @ShishKabab.

[poltak]

I followed the instructions in the recent slack thread to remove google account perms, however the login hanging issue still persists with the latest changes on feature/cloud-storage-backup. The only feedback I get is the failed req to https://memex.cloud/auth/google/callback in network tab.

I also still get that spike in CPU usage after the backup seems to complete (when processedObjects count equals totalObjects). During backup it seems fine: it's mainly network being used, which is expected.

Here's what the dump on the backup page looks like when the CPU is spiking, in case it's useful to you:

{
    "info": {
        "state": "synching",
        "totalObjects": 84,
        "processedObjects": 84,
        "collections": {
            "notifications": { "totalObjects": 2, "processedObjects": 2 },
            "directLinks": { "totalObjects": 0, "processedObjects": 0 },
            "annotations": { "totalObjects": 0, "processedObjects": 0 },
            "customLists": { "totalObjects": 0, "processedObjects": 0 },
            "pageListEntries": { "totalObjects": 0, "processedObjects": 0 },
            "pages": { "totalObjects": 30, "processedObjects": 30 },
            "visits": { "totalObjects": 45, "processedObjects": 45 },
            "bookmarks": { "totalObjects": 0, "processedObjects": 0 },
            "tags": { "totalObjects": 0, "processedObjects": 0 },
            "favIcons": { "totalObjects": 7, "processedObjects": 7 }
        }
    }
}

There is this warning error, that came up when I switched to another tab. The sidebar is disabled though, so I wonder why this error exists? @poltak

That error is specific to the sidebar stuff, not related to the backup so let's not get side-tracked in this thread. This branch is quite out-of-date with latest changes upstream so older bugs might show up still but it should be fixed in either develop or the latest hotfix branch (I will confirm that).

@mukeshkharita I saw that error as well a few times before. Again, I think it must be from the sidebar code. I will try to get it reproducible and think about a fix in a separate branch, unless @digi0ps has some time before I get around to it.

[digi0ps]

I will try to get it reproducible and think about a fix in a separate branch, unless @digi0ps has some time before I get around to it.

Actually, got it fixed in a branch I haven't sent a PR for yet. ( digi0ps/hackernews-fixes ). Will send a PR today once I add the login bug fix.

[blackforestboi]

@ShishKabab: In the Zip file you find the complete workflow for the GDrive Backup. I left commentary in red in the pages. Let me know if you have any questions about it.

Remarks:

• Between slide 3 & 4 is the signup process. People signup or login with their email and first name on slide 3 in the checkout form. Logging in at that stage should just bring them back to their cart. Ideally the signup process already logs them in, so they dont have to go through those loops here for sending themselves the email. As I understood in our call today today this would then set the cookie so that if they return to the extension they should already be logged in.
• All the credit system stuff should for now only be displayed on Wordpress.
• Sizecalculations in slide 8a are only shown when a user completed step 5 and didnt skip it.

Backup.zip

[blackforestboi]

So gave it a quick test and a couple of things were weird: 1) when doing the initial backup it showed this:

2) when i visited http://en.wikipedia.org/wiki/Memex it then spew thousands of logs to the console. Likely this is coming from the fact that i visited this page 1700 times. Still wonder why it is writing a change to every single visit in that collection

3) when I visited just that 1 page and went to the backup section again it showed this:

4) the login problem still persisist > not redirecting back to settings but still logs in 5) could not reproduce the cpu bug this time > but also working with a vanilla chrome, so no other tabs open and all other applications closed.

[ShishKabab]

Interesting stuff, thank you! About 2), did it show that without manual interaction, or did you do backup.doBackup() again? Every visit is indeed a change, and they are batched together and written to the backup.

4) is a PITA, wondering how can reproduce this myself. Any clues @poltak ?

[blackforestboi]

I only used the UI for making backups. Before that clearing Google Drive's storage.

Every visit is indeed a change, and they are batched together and written to the backup.

But should it then write 1700 visits new? i mean that it happens not by importing, but when visiting. So ideally it should only write a change for the latest visit, not for all of them gain, no?

[poltak]

4. is a PITA, wondering how can reproduce this myself. Any clues @poltak ?

Sadly, no. It's still always reproducible for me. All that shows up is a failed network req to: https://memex.cloud/auth/google/callback?code=4/bwA{base64}&scope=https://www.googleapis.com/auth/drive.appdata%20https://www.googleapis.com/auth/plus.me%20https://www.googleapis.com/auth/plus.profile.agerange.read%20https://www.googleapis.com/auth/plus.profile.language.read%20https://www.googleapis.com/auth/userinfo.profile

Have also tried different browsers, disabling extensions, etc. in case something external is messing with things, but it doesn't seem to be.

The actual backup itself seems to work though. A lot faster than previous implementation.

[ShishKabab]

i mean that it happens not by importing, but when visiting.

That's what I wanted to know :) Are you visiting pages while the initial backup is running, or afterwards?

[ShishKabab]

All that shows up is a failed network req to

How does it fail? Timeout, 404, etc.? Any other useful info?

[poltak]

It doesn't have any more information;thats the weird thing. Just sets it as red color/failed req and didnt contain any info about the response. This is in chrome network monitor. Also tried in FF but it removes that req from the network monitor less than a second after it fails, like when a new page is loaded. Maybe I can try installing Wireshark and see if I can capture anything interesting

On Fri, Oct 5, 2018, 16:04 Vincent den Boer notifications@github.com wrote:

All that shows up is a failed network req to

How does it fail? Timeout, 404, etc.?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/WorldBrain/Memex/issues/554#issuecomment-427296166, or mute the thread https://github.com/notifications/unsubscribe-auth/ABFA3O8UwjHD0RU-bqEIqXDwgYtl6YKYks5uhyCvgaJpZM4WWqh- .

[blackforestboi]

Are you visiting pages while the initial backup is running, or afterwards?

afterwards. the backup is already done. It seems to be the writing process to the changelog.

[ShishKabab]

@poltak That'd be nice :) I'll check if I can create a new test account with Google maybe leading me to the bug..

[ShishKabab]

Grumble, grumble, argh!! Just created a test account and tested login. Everything work just fine :(

[ShishKabab]

Bug reproduced on Windows :) Although Chrome doesn't give me a nice stack trace like it's 1999, the problem seems to be that the request to the Auth service's /auth/google/callback endpoint is made both from the bg script (as it should) and from the backup options page (WTF?) So, something is wrong...

[ShishKabab]

Redirect bug solved :) Now there's just the repeat visit logging issue. @poltak it seems that upon each visit to a page, all previous visits are also logged. Are the previous visits also modified by any chance? If not, maybe we're storing the pk in the change table the wrong way?

[poltak]

@poltak it seems that upon each visit to a page, all previous visits are also logged. Are the previous visits also modified by any chance?

Yes, after having a look this was indeed the case. I have updated this behaviour in https://github.com/WorldBrain/Memex/commit/bbba008c602b759035e775764468a226a7924417 to do a comparison between the current and existing visits in the DB before choosing to do the write. Let me know if the issue persists or if there's any other issue with the change

[blackforestboi]

@poltak Problem is gone :)

[ShishKabab]

Whoohoo! :)

[blackforestboi]

Backup.zip

[ShishKabab]

Implemented and released :)