search

kiwibrowser / src.next

Source-code for Kiwi Next, a Kiwi Browser auto-rebased with latest Chromium
BSD 3-Clause "New" or "Revised" License
2.26k stars 287 forks source link

Backup System #122

Open NotesOfReality opened 6 years ago

NotesOfReality commented 6 years ago

Could you please add as a new feature a backup/restore system? I would like it to mainly backup opened tabs, web navigation history, bookmarks and chrome flags and maybe settings. The possibility to choose what part of the backup may be restored and to view its content would be useful. The backup should be made by creating a file (I guess it should be a text type) and nothing more.

At anyway your speed hacks are crazy, my best compliments because this is the snappiest chromium build I've tried since maybe four years of various experiments with other chromium and non-chromium browsers.

gdgsdg123 commented 5 years ago

history: '/data/data/com.kiwibrowser.browser/app_chrome/Default/History'

bookmarks: '/data/data/com.kiwibrowser.browser/app_chrome/Default/Bookmarks'

chrome flags & settings: '/data/data/com.kiwibrowser.browser/app_chrome/Local State' '/data/data/com.kiwibrowser.browser/app_chrome/Default/Preferences' '/data/data/com.kiwibrowser.browser/shared_prefs/com.kiwibrowser.browser_preferences.xml'

NotesOfReality commented 5 years ago

Thank you very much for the precious information, if only I had known a bit of programming languages I would have implemented the backup system by myself and then created a pull request.

gdgsdg123 commented 5 years ago

Just manually back them up/restore with any adequate explorer, actually...

NotesOfReality commented 5 years ago

Yeah, I already know that, I simply think that adding such a feature with a proper UI could make the app go in the top 5 or something like that

madranet commented 5 years ago

Thanks. This is good for helping back up an existing Kiwi config and restoring it. But doesn't address importing data from another browser.

Please don't fall into the trap so many developers do of bringing out a new product and expecting people to migrate to it without being able to bring their existing data over from your rivals, who they've been [more or less] happily using for years.

Doesn't directly affect me as I have my bookmarks stored on pinboard. But a lot of folks will have years worth of bookmarks and history stored in $CurrentBrowser and are much more likely to move to $NewBrowser if they can bring that stuff with them.

NotesOfReality commented 5 years ago

Thanks. This is good for helping back up an existing Kiwi config and restoring it. But doesn't address importing data from another browser.

Please don't fall into the trap so many developers do of bringing out a new product and expecting people to migrate to it without being able to bring their existing data over from your rivals, who they've been [more or less] happily using for years.

Doesn't directly affect me as I have my bookmarks stored on pinboard. But a lot of folks will have years worth of bookmarks and history stored in $CurrentBrowser and are much more likely to move to $NewBrowser if they can bring that stuff with them.

I think this is a quite worth of attention denotation, it now is so more than ever since Kiwi Browser developer even has brought Chrome extensions support.

NotesOfReality commented 5 years ago

In case anybody may ever need it, here I leave the "opened tabs" path, which has recently been tested by me.

opened tabs: '/data/data/com.kiwibrowser.browser/app_tabs/' copy the whole folder and replace the app_tabs/0/tab_state0 file or edit it adding the URLs listed there

Moreover I'd say we don't need the bookmarks path since we have the bookmark im/export feature.

gdgsdg123 commented 5 years ago

Moreover I'd say we don't need the bookmarks path since we have the bookmark im/export feature.

Prefer doing so by modifying the files anyway...

DUOLabs333 commented 3 years ago

While this may work, it does require you to have root. I tried using the "Extensions" tab to manually go inside its folder (since it would have access), but it didn't work.

elig0n commented 2 years ago

Is there a way to backup the extensions list or the extensions with their configs collectively ?

DUOLabs333 commented 2 years ago

An (relatively) easy way this could be implemented is to save all your data to /sdcard/Kiwi Browser. Then there would be no need to implement a backup system, as you could just copy the folder to wherever you like.

dausruddin commented 1 year ago

Thank you @gdgsdg123 and @NotesOfReality. Luckily I have root.

For saved passwords, /data/data/com.kiwibrowser.browser/app_chrome/Default/Login Data

Vojtak42 commented 1 year ago

An (relatively) easy way this could be implemented is to save all your data to /sdcard/Kiwi Browser. Then there would be no need to implement a backup system, as you could just copy the folder to wherever you like.

This would be really useful.

Vojtak42 commented 1 year ago

history: '/data/data/com.kiwibrowser.browser/app_chrome/Default/History'

bookmarks: '/data/data/com.kiwibrowser.browser/app_chrome/Default/Bookmarks'

chrome flags & settings: '/data/data/com.kiwibrowser.browser/app_chrome/Local State' '/data/data/com.kiwibrowser.browser/app_chrome/Default/Preferences' '/data/data/com.kiwibrowser.browser/shared_prefs/com.kiwibrowser.browser_preferences.xml'

But without root you don't have access there. Please implement feature like Termux has so you can access all of it's files even without root

Ibuprophen commented 1 year ago

This appears (to me) very similar to the https://github.com/kiwibrowser/src.next/issues/728 issue I had submitted quite some time ago without any resolution thus far.

~Ibuprophen

zpangwin commented 1 year ago

Since #845 was just closed with the message:

Closing and linking this to #122

I would like to request that whatever backup method is added also support exporting to a browser-neutral format (such as .html or txt) for the currently opened tabs as the main ask in #845 was to be able to have something that could allow exporting currently opened tabs from kiwi in a format that could be opened on desktop browsers without requiring cloud services.

HT-7 commented 1 year ago

Very important. Also necessary for switching from a non-rooted to a new phone. Without proper data portability, it is de-facto impossible to bring the tabs and history and bookmarks and (depending on browser) saved pages to a new phone.

Individually copying and pasting a thousand URLs or depending on cloud services that can go defunct any day without warning are obviously impractical.

I believe that one can only really own a smartphone, and a tablet for that matter, if it is rooted. Without root access, there are so many crippling restrictions, especially regarding file access. The data can not even be backed up properly! And backups are highly important!

https://beepb00p.xyz/sad-infra.html

Nichtraucher commented 9 months ago

I'm not sure why the devs decided to not allow backups. If data safety is a concern, there is a way to back up data safely if it's encrypted. .

Many thanks to the developers for their efforts! :-D

EDLLT commented 9 months ago

In case anybody may ever need it, here I leave the "opened tabs" path, which has recently been tested by me.

opened tabs: '/data/data/com.kiwibrowser.browser/app_tabs/' copy the whole folder and replace the app_tabs/0/tab_state0 file or edit it adding the URLs listed there

Moreover I'd say we don't need the bookmarks path since we have the bookmark im/export feature.

I have looked into the files. They have some gibberish when I tried using cat directly. How could I extract the URLs only from that Also I noticed that there were two directories. Namely, 0 and custom_tabs

lrq3000 commented 6 months ago

FYI, both bookmarking all tabs and sharing all tabs as a simple text list are implemented in Brave for Android (I tested, both work). It seems this is simply because Brave merged in the latest chromium release. Note that exporting to bookmarks seem to work with an arbitrary number of tabs, whereas exporting to a list is limited and may crash (likely because of a memory overflow since it attempts to store in the clipboard). Brave does not support extensions.

Until an on-device solution is implemented by the developer, I think the most reliable non-root approach is by using the remote debug tools (but note I did not try this myself, but I can't see why it would not work).

lrq3000 commented 6 months ago

FYI I could get a list of opened tabs using the method described here. Summary:

/EDIT: this only displays the recent tabs of the current session that are still in-memory, this does NOT solve the issue of exporting past tabs that are not loaded in-memory.

Here are a list of extensions that can do the same (exporting currently opened tabs) that work in Kiwi:

lrq3000 commented 6 months ago

After some fiddling, I found the really working and future-proof way to export really all tabs.

The future-proof non-root method to export all tabs from Kiwi browser, including past, memory-unloaded ones, is to enable Developer Mode on the Android phone, then enable USB Debugging on the phone, then connect it to a computer via USB, and then on the computer launche Chrome and open the page chrome://inspect/#devices , then tap on the phone to enable USB debugging for this session, and after a few seconds you should see a list of all your tabs, including ones in private session / incognito mode, and you can simply copy/paste everything as plain text (there are both the pages titles and URLs). This method works for all chromium based browsers, and is described in more details here and as first mentioned by another Kiwi user here. Note that it is likely that this method only works if you have installed a "dev" release of kiwi, with the debug flags enabled (I only tried with a dev release).

Now if Kiwi could just get synced with the latest chromium for Android features, it would support selecting all tabs in the tabs menu and then bookmarking/export as Brave and Vivaldi browsers can. But personally this solution will allow me to use Kiwi in the meantime as I can backup all my tabs this way, even if it is a bit cumbersome.

Nichtraucher commented 6 months ago

There is (perhaps) a backup method - limited to Android versions 11 and below though. I didn't quite figure it out and asked about it on reddit here Perhaps some readers here are able to chime in and explain carefully in small steps how this modification can be done successfully?

lrq3000 commented 5 months ago

@Nichtraucher I did not know about this flag, I'm not sure it really exists since a long time, the post that was linked to is from 2013. Either way, it is unfortunately not a future-proof or even present-proof method, since modern Android releases make it very difficult for apps to store their data outside of their isolated folder, and its very difficult to access them, you need root or to use adb. A flag is certainly not sufficient anymore to allow external access to an app's files, I know I worked on several apps to implement backup features that got broken since Android 10.

vlasky commented 2 months ago

To export all the URLs of open browser tabs in Kiwi on your Android phone using the adb utility on Windows, you can follow these steps.

I tested this procedure with Kiwi Browser running under Android 14 on my Samsung Galaxy S22 Ultra, connected via USB to my Windows 10 PC.

Prerequisites:

  1. Android Device: Ensure USB debugging is enabled on your Android device.
  2. ADB Installed: Ensure you have the Android Debug Bridge (ADB) installed on your Windows computer.
  3. Kiwi Browser Installed: Ensure Kiwi browser is installed and running on your Android device.
  4. USB Cable: Ensure your Android device is connected to your Windows computer via USB.

Steps:

  1. Enable Developer Options and USB Debugging on Android:

    • Go to Settings > About phone.
    • Tap on Build number seven times to enable Developer options.
    • Go to Settings > System > Developer options.
    • Enable USB debugging.

  2. Connect Android Device to Windows Computer:

    • Connect your Android device to your Windows computer using a USB cable.
    • Allow USB debugging authorization on your Android device when prompted.

  3. Open Command Prompt on Windows:

    • Press Win + R, type cmd, and press Enter.

  4. Verify ADB Connection:

    • In the Command Prompt, type: 
      adb devices
    • Ensure your device is listed. If not, troubleshoot the USB connection and authorization.

  5. Use Chrome Remote Debugging:

    • Open Kiwi browser on your Android device.
    • In the Command Prompt, type: 
      adb forward tcp:9222 localabstract:chrome_devtools_remote
    • Open a web browser on your Windows computer and go to: 
      http://localhost:9222/json
    • This will display a JSON file containing information about all open tabs in the Kiwi browser.
    • Copy the content of the JSON file displayed in your web browser.
    • Save it as a file named kiwi_tabs.json.

  6. Extract URLs from JSON:

    • There are many ways you can extract URLs and page titles from the JSON file. Here is a Windows PowerShell script that will do the job: 
      
# Load the JSON file
$json = Get-Content -Raw -Path "kiwi_tabs.json" | ConvertFrom-Json

Filter out entries where 'id' is not a numerical value and sort by 'id' numerically

$filteredSortedJson = $json | Where-Object { $.id -match '^\d+$' } | Sort-Object { [int]$.id }

Initialize a counter for the incrementing number

$counter = 1

Process each entry and format the output

$output = $filteredSortedJson | ForEach-Object { "$counter" $.title $.url "" $counter++ }

Output the result to the console or save to a file

$output | Out-File -FilePath "kiwi_tabs.txt" -Encoding utf8


### Explanation:
1. **Load the JSON file**: The script reads the JSON file and converts it into a PowerShell object using `ConvertFrom-Json`.

2. **Filter and Sort**: It filters out entries where the `id` is not numerical using a regular expression (`-match '^\d+$'`). Then it sorts the remaining entries numerically by `id`.

3. **Auto-increment Counter**: A counter is initialized to start from `1`, which will be used to number the entries.

4. **Process Each Entry**: The script processes each entry, outputting the incrementing number on the first line, the title on the second line, and the URL on the third line. An empty line is added after each entry.

5. **Output**: The result is saved to a file named `kiwi_tabs.txt` with UTF-8 encoding.

### Running the Script:
1. Save the script in a `.ps1` file, for example, `process-json.ps1`.
2. Run the script in PowerShell:
   ```powershell
   .\process-json.ps1

This will create an kiwi_tabs.txt file in the same directory, with the following format:

1
Page Title 1
URL 1

2
Page Title 2
URL 2

The entries with non-numerical id values will be skipped, and the remaining entries will be numbered and formatted as specified.

  1. Finishing up: -To stop ADB forwarding, you can use the adb forward --remove command followed by the local port you forwarded. Here is how you can do it:

  2. Open Command Prompt:

    • Press Win + R, type cmd, and press Enter.

  3. Remove the Forwarding:

    • To remove the specific port forwarding you set up (e.g., tcp:9222), type: 
      adb forward --remove tcp:9222
    • If you want to remove all port forwarding, type: 
      adb forward --remove-all

These commands will stop ADB from forwarding the specified ports, effectively stopping the remote debugging connection you established earlier.

lrq3000 commented 2 months ago

@vlasky did you try yourself the instructions you gave? So far all approaches involving adb only exported opened tabs during the current session, but not tabs open during previous sessions when the browser got closed after, before the attempt to backup. I have found no way to achieve this so far on a non-rooted phone, it seems newer versions of Android isolate apps databases including past browser history, even from browser extensions, maybe as a way to safeguard privacy.

vlasky commented 2 months ago

@vlasky did you try yourself the instructions you gave? So far all approaches involving adb only exported opened tabs during the current session, but not tabs open during previous sessions when the browser got closed after, before the attempt to backup. I have found no way to achieve this so far on a non-rooted phone, it seems newer versions of Android isolate apps databases including past browser history, even from browser extensions, maybe as a way to safeguard privacy.

@lrq3000 I certainly did. I exported 3000 tabs.

Vojtak42 commented 2 months ago

@vlasky did you try yourself the instructions you gave? So far all approaches involving adb only exported opened tabs during the current session, but not tabs open during previous sessions when the browser got closed after, before the attempt to backup. I have found no way to achieve this so far on a non-rooted phone, it seems newer versions of Android isolate apps databases including past browser history, even from browser extensions, maybe as a way to safeguard privacy.

I used auto clicker to swipe throught all the tabs and then exported it using session buddy.

lrq3000 commented 1 month ago

I confirm @vlasky's method works on non-rooted phones, and that it can export an arbitrary number of tabs instantaneously, without requiring to load all tabs in memory beforehand with an autoclicker (this would be impossible with more than 1K tabs!).

I just exported 2600+ tabs.

The only downside is that tabs groups are not saved, but tabs are saved in chronological order (after being filtered by @vlasky 's script).

Thank you so much @vlasky !

@d3ward Maybe this can be added to the documentation? Indeed this method to backup all tabs looks extremely reliable and robust since it works on all devices, even non-rooted ones, and can export an arbitrary number of tabs without first loading them in memory.