search

encryptic-team / encryptic

An encryption-focused open source note taking application
Mozilla Public License 2.0
348 stars 40 forks source link

Change and Document how to migrate from Laverna #58

Open salessandri opened 5 years ago

salessandri commented 5 years ago

Motivation

Transition from Laverna to Encryptic wasn't seamless. In fact, I haven't been able to successfully migrate my Laverna generated backup into Encryptic.

I also couldn't find a description of how to do that migration. E.g. should I use the import from previous device? Or should I create a profile and then import the data from the settings page?

Proposal

I propose to treat Laverna generates backups as an external application. Similarly as Evernote is treated.

If that is the case, Laverna would have its own import button in the settings page and that would be the only place where those backups are accepted. Any other place where a Laverna backup is uploaded it should be rejected.

As an added change I would be somewhat radical and establish that only data (tags, notebooks, notes) will be imported from Laverna backup. No config information at all. This will make it easier to reconcile things, specially in the future where configuration options might diverge a lot.

User Workflow

To import from a Laverna backup a user would have to:

  1. Create a normal Encryptic profile.
  2. Go to the settings page, backup section.
  3. Click on Import from Laverna
  4. The user will be asked to pick a zip file.
  5. The user will be shown an "Importing in Progress" while the importation process takes place
  6. The user will be shown an "Import Done" with an OK button.
  7. When the user presses this button will be redirected to the Settings page, Backup section.

If any error happens during step 5, step 6 would be an error page explaining the error found.

Caveats

daed commented 5 years ago

It is a dodgy process. I managed to convert mine, but it didn't seem to work every time. I've looked at and made changes to the import/export code several times. It's a mess (it was pretty bad even before I mucked around in it) and I'd almost like to overhaul it completely, but, and this is partially my fault too, there are at least five different means of importing and I'd like to not completely break backwards compatibility for any user currently on any version.

Known import methods

1a. Laverna v0.7.51 backup file via backup -> "recover from backup" 1b. Laverna v0.7.51 direct import from notes-db in your browser's web storage

  1. Laverna dev branch backup file via backup -> "recover from backup"
  2. Encryptic backup file during profile creation
  3. Encryptic backup file via backup -> "recover from backup"
  4. Evernote backups

It's technically not a full import, but you can also import just your key during profile creation as well. (thanks @salessandri for fixing that btw!)

Number 1a and 1b SHOULD use the same code to import, with 1a having the additional step of reading the zip file, but I'm not sure they are. To make matters more confusing, 1b doesn't actually give you any prompt for import until you create an Encryptic profile and log in for the first time. I'd almost like to just remove 1b as an option altogether.

Number 2, 3, and 4 I think use the same code to import, except I had to change some things like looking for a "encryptic-backup" directory rather than "laverna-backup" in the zip file.

Proposed import methods

I just want to make sure I understand. Your proposal would give us the following:

  1. Laverna (any version) backup file via backup -> "Import laverna"
  2. Encryptic backup file during profile creation
  3. Encryptic backup file via backup -> "recover from backup"
  4. Evernote backups

If so, I would have no objections.

Configs

I like the idea of discarding any config that isn't an Encryptic config. That will reduce headaches later. It's a little offtopic, but I'm a little worried about how to handle an Encryptic config as it changes, like if features are added/removed/changed.

Do we like the idea of a version number in the config, and then we try to migrate it to the newest version when it's upgraded?
Do we just "try really hard" to import it when the time comes and if any part of it breaks, we accept defaults?

Addressing Caveats

Duplicates

I believe that notebooks/tags are associated by GUID, not name, so we could have three "test" notebooks, each with different contents. I don't know what current behavior looks like. The options I see:

  1. Do nothing. This results in the user potentially having two of every notebook and they must go through and reorganize. This will probably be recognized by users as a "bug".
  2. Attempt to combine notes associated with the duplicate notebooks/tags into the existing versions of them. This would probably be the most complicated but would be the most intuitive to users.
  3. Check for existing notebooks and tags already named the same as the ones being imported, and any new notebooks/tags we rename to an unused name and proceed. Duplicate notebook "test" would get renamed to "test-import" or "test-1" or something like that. This results in the user still needing to potentially reorganize, but at least they can tell the old notebook from the new one, and it is expressed in a way that would probably not appear to look like a bug.

I like number 3 myself. I am open to alternative suggestions.

Progress and statistics

I can't think of any reason not to display this. This might also be a good place for debugging/error logging.

salessandri commented 5 years ago

Number 1a and 1b SHOULD use the same code to import, with 1a having the additional step of reading the zip file, but I'm not sure they are. To make matters more confusing, 1b doesn't actually give you any prompt for import until you create an Encryptic profile and log in for the first time. I'd almost like to just remove 1b as an option altogether.

Completely agree with this. I would remove method 1b. It's too error prone and requires the user to touch internals.

I just want to make sure I understand. Your proposal would give us the following:

  1. Laverna (any version) backup file via backup -> "Import laverna"
  2. Encryptic backup file during profile creation
  3. Encryptic backup file via backup -> "recover from backup"
  4. Evernote backups

I would remove method 2. I would rather have the user create a profile with the appropriate key and then use method number 3. The workflow is more straight forward and easier to document.

Configs

I like the idea of discarding any config that isn't an Encryptic config. That will reduce headaches later. It's a little offtopic, but I'm a little worried about how to handle an Encryptic config as it changes, like if features are added/removed/changed.

Do we like the idea of a version number in the config, and then we try to migrate it to the newest version when it's upgraded? Do we just "try really hard" to import it when the time comes and if any part of it breaks, we accept defaults?

In my experience the best way to handle this case is to have the "current storage version" stored somewhere in the db (can be part of the configs store).

When the user logs in, the first step will be to apply all the necessary migration logic to upgrade the storage version to the current one. The easiest way to do this is to basically keep all the migration logic from version to version and apply them one by one. For example: current storage version is 2 when the target version is 5, then we apply 2 -> 3, 3 -> 4 and 4 -> 5.

That guarantees that if a user hasn't opened the app in a long time will transition the path as if he had. And it makes it easy for us to maintain and not have to think about what if the user was in version X and now jumps to version Y?

Regarding import of config, I would apply the aforementioned logic and probably override the current values. I am assuming import from backups is not a thing you would normally do.

Regarding duplicates, I would probably try to go for option 2 if possible. It shouldn't be that hard. Otherwise, option 3 is my next choice.

daed commented 5 years ago

I've finally got back to working on this. I've removed 1b on my local branch. Current import methods are:

  1. Laverna v0.7.51 backup file via backup -> "recover from backup"
  2. Laverna dev branch backup file via backup -> "recover from backup"
  3. Encryptic backup file during profile creation
  4. Encryptic backup file via backup -> "recover from backup"
  5. Evernote backups

With other changes made to migration, a test of method 1 worked for me. I'm going to rewrite the button to be a little more explanatory.

Current plans are to combine 1/2 into an "Import laverna" button and consolidate legacy migration code further away from encryptic import/export. Method 3 will likely be removed as there is a means to import your key during profile creation, and then you can import encryptic after you already have the profile generated. We have a means for changing your passphrase after profile generation, but we don't have one for importing a new key to an existing profile. I kind of feel like that's something we need to consider also. I suppose we would need to decrypt notes using the old key, and then encrypt them again with the new one.

I don't honestly even know if the Evernote option works. I suppose I'll have to dust off my account and try it.

daed commented 5 years ago

I've made some improvements on this. "Recover from backup" should successfully handle both laverna and encryptic. There was an issue in the way models were being handled during the encryption/import stage that was potentially impacting anything being imported. The import methods still haven't been standardized to what we talked about above, but this was I think a major step toward being able to do so.