Currently, there's no way to persist data between evaluations of a project, meaning there's no way to build projects that have a memory. For example, if one wanted to create a to do list app, there would be no way, because every time the project evaluates, it would start from scratch for its initial values.
What's the design idea?
The idea is to use Source files as a database. In addition to typographic and audio output, have an API for creating or overwriting a source file by name. This would allow for all kinds of stateful applications like data science, generating data tables, for example.
Who benefits?
Anyone wanting to create an application with a memory.
Design specification
The API would be a single structure definition with two inputs:
value is the data value to render as a source file, which can be any of the built in data types
When this structure is created and returned as part of the program's value, if the source file with the name doesn't exist, it will be created with valid Wordplay code corresponding to the value , and if it does the source file's text will be updated. In either case, the change to the project source will trigger a project re-evaluation. Applications that read the source file (presumably all of them) will get the new value, allowing application state to be re-rendered.
By varying names, projects can have multiple source files corresponding to different data structures.
Note: the final value of a program is usually output like a Phrase, Group, or Stage, but we would extend this to allow return values to be lists of output, like this:
[ Phrase('hi') Source('data' [1 2 3])]
This would show Phrase but also write [1 2 3] to a source file named 'data'.
There are a few exceptions possible when creating a source:
If name corresponds to any source file that is not a single expression that evaluates to a literal, it will evaluate to an exception, to avoid overwriting any code
If the name is an empty string, it will evaluate to an exception, since there will be no matching source file to write to.
If the value is too big for a source file (primarily determined by limits on the size of Firestore documents), it will fail, as the source will not be able to be saved to the database. This puts some inherent limits on how much can be stored in an application.
If a project has too many source files, it will evaluate to an exception, to avoid creating an infinite number of sources for a project (which would probably crash the UI).
There are a few performance things to address:
There may be a need to improve the performance and usability of projects with a very large number of source files.
We may want to avoid persisting revised projects immediately after they are updated, as this would trigger a large number of database writes. Verify that we aren't aggressively persisting.
There are also a few limitations this places on the kinds of data persistence that's supported. There are generally four types of applications creators might want to make:
Entirely personal projects where its okay to modify project source because it’s private
Collaborative where there’s shared data, but still private to a group
Public projects where data is global but still accessible (e.g., crowdsourcing)
Public projects where data is per creator and private (e.g., games)
Types 1 and 2 are supported by the above design, because everyone has write access to the project. Number 3 is not supported, because public projects are still read only, and there's no way to make a public project data write only (and that would be quite risky!). Type 4 is a common use case that wouldn't be well supported by this, but that's okay, because we're not trying to make an app platform to reach large numbers of people. And there is a workaround for type 4: copying a project and creating personal data in the copy.
amyjko
commented
5 months ago
What's the problem?
Currently, there's no way to persist data between evaluations of a project, meaning there's no way to build projects that have a memory. For example, if one wanted to create a to do list app, there would be no way, because every time the project evaluates, it would start from scratch for its initial values.
What's the design idea?
The idea is to use
Sourcefiles as a database. In addition to typographic and audio output, have an API for creating or overwriting a source file by name. This would allow for all kinds of stateful applications like data science, generating data tables, for example.Who benefits?
Anyone wanting to create an application with a memory.
Design specification
The API would be a single structure definition with two inputs:
•Source(name: '' value: '' | `` | # | ? | [] | {} | ⎡⎦)nameis the name of the source file to overwritevalueis the data value to render as a source file, which can be any of the built in data typesWhen this structure is created and returned as part of the program's value, if the source file with the name doesn't exist, it will be created with valid Wordplay code corresponding to the value , and if it does the source file's text will be updated. In either case, the change to the project source will trigger a project re-evaluation. Applications that read the source file (presumably all of them) will get the new value, allowing application state to be re-rendered.
By varying names, projects can have multiple source files corresponding to different data structures.
Note: the final value of a program is usually output like a
Phrase,Group, orStage, but we would extend this to allow return values to be lists of output, like this:[ Phrase('hi') Source('data' [1 2 3])]This would show
Phrasebut also write [1 2 3] to a source file named 'data'.There are a few exceptions possible when creating a source:
namecorresponds to any source file that is not a single expression that evaluates to a literal, it will evaluate to an exception, to avoid overwriting any codeThere are a few performance things to address:
There are also a few limitations this places on the kinds of data persistence that's supported. There are generally four types of applications creators might want to make:
Types 1 and 2 are supported by the above design, because everyone has write access to the project. Number 3 is not supported, because public projects are still read only, and there's no way to make a public project data write only (and that would be quite risky!). Type 4 is a common use case that wouldn't be well supported by this, but that's okay, because we're not trying to make an app platform to reach large numbers of people. And there is a workaround for type 4: copying a project and creating personal data in the copy.