search

willhlaw / node-firestore-backup-restore

Google Firebase Firestore backup and restore tool
91 stars 24 forks source link

Can we backup first and then restore from local folder? #3

Closed fyarepo closed 6 years ago

fyarepo commented 6 years ago

Can we backup first and then restore from local folder?

willhlaw commented 6 years ago

Not as of v1.1.2, but PR welcome.

Currently, the loop writes documents to disk for backup and simultaneously sets documents in the restore firestore for restore.

The approach to take is to detect if --backupPath and --restoreAccountCredentials were provided but --accountCredentials was not. Then rather than going into the accountDb.getCollections() call here, you would instead read from the filesystem.

MrDrProfX commented 6 years ago

This feature would be great. I'm afraid I don't understand what restore is supposed to do currently. Restore what? The command takes in two firebase app credentials and a destination backupPath. Not sure what could be restored given these args.

willhlaw commented 6 years ago

You can restore all docs from one firestore db to another. For instance, you may want to backup from "dev" and migrate the docs to "staging". Perhaps "upload" is a better term than "restore" in this case.

With some improvements, backup and restore can be separated so that one could backup "dev" to disk, work on a feature that happens to screw up the docs somehow, and then restore from disk to "dev".

So here are the use cases I see:

With that said, perhaps there could be a better name for the library that can perform the above.

Any suggestions?

rtman commented 6 years ago

I think a better name for migrate from one db to a second db is Clone

The docs on this package are confusing and I had to come here to really understand what this package does. Perhaps you could rewrite the docs with examples on how to do the following:

  1. backup to disk
  2. Clone db to a second db
  3. Upload from disk (not yet available)

Any ideas on when we might have Upload from disk? This could be very useful for uploading a large dataset to firestore for data entry or restoring a back up.

pmarconi commented 6 years ago

I'm interested on this proposed feature, and I'm wiling to take this on.

I think we should be able to use only the flags -B and -a2 to upload from local disk to a Firestore db. Currently, it's not possible use this project without defining the flag -a (accountCredentials), this flag should't be required anymore

So, to 3. Upload from disk (not yet available), we should use firestore-backup-restore --backupPath /backups/myDatabase --restoreAccountCredentials path/to/restore/credentials/file.json

How does that sound? If it sounds good, I will start working on it,

willhlaw commented 6 years ago

That sounds great, thanks @pmarconi. Can you take a stab at updating the readme as well?

I’ll follow up on any improvements you make with the docs to see if we should name each scenario something different - such as @rtman proposed “clone” for this feature.

MrDrProfX commented 6 years ago

@pmarconi Awesome. Something I'm wondering about is what will happen to new documents/fields and collections that have been added since the backup. Two options could be:

  1. Replace: delete the entire target firestore db and completely replace it with the disk backup
  2. Merge: merge the target firestore db with the disk backup

Perhaps this could be an additional command line argument. Personally I would rely on the merge for my project.

willhlaw commented 6 years ago

Good idea @MrDrProfX.

  1. Issue #7 created to add a delete command and replace feature.
  2. Issue #8 created to add the restore with merge feature.

This feature here to restore from disk should stick with the existing behavior of the restore function which is to set (overwrite) the document. Keeping these features in distinct pull requests will make it easier.

pmarconi commented 6 years ago

Ok, I'll consider those options

One of the problems that I have, is that references to documents and GeoPoint are objects that once restored in firestore, from our backup, are stored as objects and not as references or GeoPoints. For Dates too, are restored as strings

Then I think it will be necessary to store in our backup the data and the types, specified in https://firebase.google.com/docs/firestore/manage-data/data-types to then save them and treat them accordingly

pmarconi commented 6 years ago

I create the following PR #10 , only for this feature

willhlaw commented 6 years ago

Thanks for @pmarconi, this feature request can be closed. See #10. Upgrade to firestore-backup-restore@1.2.1.

Now, by default, documents that are backed up to disk have each field's value stored in an object {value, type}. The type information is used when restoring later to maintain full fidelity. If you want to backup without the type information, use the -J or --plainJSONBackup option.