[super] Persistence of projects and their associated materials

[rufuspollock]

Note there is a distinction (probably) between:

1. Persisting Data Explorer config + list of projects + other user specific info (e.g. github login token) - #51
2. Persisting individual projects

We are focusing on the second item here: persisting a given project

Why

Why do we want persistence?

• For a given user they want to know their work will be saved and not lost
• With versioned storage we could have reversion (and undo)
• Sharing work with others - if we persist in a way that allows others to access

  Implementation

There are several aspects

1. What we persist
2. How we serialize
3. Where we persist to (i.e. beyond memory)

   What we persist

   • Project metadata
   • Scripts
   • Views
   • Data(sets)

Do we also need to support persisting intermediate changes so we can undo quickly?

How we serialize

Not sure we need to specify here but obviously stuff that can be represented as JSON or CSV (+ datapackage).

Where we persist to

We will likely persist to two places:

• Short-term: localStorage (at least for some info) + indexed db or filestorage for larger material
  • This is temporary storage that allows for quicker interaction and can be seen as a form of cache
• Longer-term: Gists (or possibly full git repos on github)

Notes:

• private gists cannot be made public - you will have to repost => let's focus on public only for the present
  • Actually private gists are only private in the sense others cannot see them if they browse your gists - but if you have the id anyone can see them

We will focus on Gist persistence first as it gives us ability to share with others

Gist persistence

http://developer.github.com/v3/gists

How do we save structure of the project? Options:

• All in one: put everything into one big blob of JSON and save (that's the current approach)
• Split out scripts, views and datasets to separate files
  • means these could be edited in a normal editor
  • separate updates of specific files (?)
  • easier to view

Separate files

• datapackage.json (includes dataset atm - but should be split out)
• scripts/...
• datasets/file.csv

[rufuspollock]

Closing as nothing major left to do in the near term. Definitely need to think further about short-term versus long-term persistence but basic setup is now working.