This post is intended to clarify and set the direction of where Scriptchat (SCRIPTCHAT?) is going. This reflects roughly my thoughts at the moment but is still early enough to be open to discussion and interpretation.
Architecture
This is a client-side only single-page app bootstrapped with create-react-app. It's React/Redux-based, which is my go-to framework of choice, in a "If all you have is React/Redux then every problem looks like a nail" sort of way. I am open to reasonable proposals to adopt another framework if a sufficient case can be made.
There is no server-side component. An early experiment of this tried a server-side component as its own chat server, but this seems like overkill and you would never get people to use it, which is why we use Slack transcripts now. Once we're only parsing existing logs, this can all be done client-side, which means (a) hosting is free (b) we don't need to worry about keeping private information safe over the wire or on the back-end side. Also, since the view will eventually be in-browser, this simplies the user interaction dramatically, as we won't require users to download, install, or run any command lines things to check it out. All you need is access to the exports from a Slack channel.
User flow
It basically looks like this:
Add Slack logs.
Get a script.
2b. Possibly tweak it a little.
Export / save it somewhere, if you want.
1. Add Slack logs.
Slack logs can be obtained by any owner of a Slack channel (see issue #2). At minimum, one would need two files: (1) the transcript itself (a JSON file), and (2) the users.json file, which contains data about all the users in that channel. Putting these two together is sufficient to generate a screenplay.
Transcripts are broken up by channel, then by day. We may need to think through how to handle screenplays that span multiple days.
The users file contains information about the name of the user. It doesn't contain other things that screenplays usually need to know like gender or physical description. We may need to think through how to add or infer that information.
The formats are unique enough that Scriptchat should be able to understand what data is being added. There should only need to be one "add a file" button (or drag-drop onto the window to add) -- you wouldn't need separate buttons for each type. Scriptchat may either decide to maintain a collection of different files or just overwrite the previous one when a new one is added.
Providing helpful errors is a good idea at this stage:
When a JSON file being dropped cannot be understood as either a chat log or users file.
When a chat log and users file doesn't seem to match.
Reject any file that cannot be read as text (e.g. binary).
Text files may be Fountain based, in which case we pass it straight through to Fountain for display, since we get it "for free." This makes it easy for an exported script file to be loaded through the same interface later.
2. Get a script.
Once the two Slack files are uploaded, processing happens. This step is entirely done in code. Data in, data out. This pipeline is good for anyone who knows JavaScript but doesn't want to meddle with the browser part of things. It is especially good if you like functional programming and writing tests. This is our "secret sauce," as the quality of this pipeline influences the quality of the screenplay.
Roughly, these things need to happen.
Process users.json:
Each user is an object in an array. This means looking up any user needs to iterate over the array and check the id property; this could get expensive. If we rewrite the data structure so that the id is a key lookup, matching users to the transcript gets easier to do.
Surface the useful information: first name and last name is the most important right now. Slack does not seem to use the same properties for everyone (it depends on what data the user provided) so we need to do some normalization.
Not all information is relevant and it may be dropped.
Convert chat log to Fountain tokens. Fountain.js uses a token structure internally to handle the translation between parsing a Fountain-formatted text file and the HTML view. It is not a public API but because it works, we should steal it. Holding our data state in this token structure allows us to work on it programatically.
Process tokens:
User IDs need to be replaced with real names. Use CAPITAL LETTERS and full names (if we have it) when a character is first introduced in any action. Use just the first name when spoken by another character.
Consecutive messages from the same user should be consolidated.
Emojis that are present and look like emotion markers, rather than grammatical entities, should be extracted and converted to parentheticals or actions.
People entering or exiting the channel are actions.
We can really get super creative here; this is just the obvious ones, but imagine what else you can do here.
Output tokens. When processing is done, the tokens are shuffled off to a place in the Redux store, where the React view takes over.
Convert tokens to HTML markup. This is entirely ripped from Fountain.js and really needs very little work from us.
Also, title page
There is a special part of the Fountain syntax reserved for the title page. The view is essentially taken care of, but to populate it, there is a set of key-value pairs that make it up. It's possible that in the parsing stage we can infer some of these, but it's also easier on the user to expose the UI for it so that it can be customized.
In addition to editing title page properties, it's possible that the script itself might need some tweaking. Earlier, I mentioned that it would not be possible to obtain gender pronouns from Slack. We may need to provide UI after the fact to add those to users, plus other identifying information or characteristics that would enliven the script.
This step is extremely open to interpretation. Also, we may not know exactly what tweakable knobs we need to present until doing a number of test runs to figure out what really needs some human intervention.
3. Export / save it somewhere.
Once you have a script, it should go somewhere. Exporting as Fountain format is a must, since it can be read by other screenplay writing tools like Final Draft. Exporting as PDF is just as important, because that's easier to send around. Generating either of these can be done by using the token structure we have and writing that to a file.
And that's Scriptchat! In a future phase, we put on an actual production!
Laurian
commented
6 years ago
This post is intended to clarify and set the direction of where Scriptchat (SCRIPTCHAT?) is going. This reflects roughly my thoughts at the moment but is still early enough to be open to discussion and interpretation.
Architecture
This is a client-side only single-page app bootstrapped with create-react-app. It's React/Redux-based, which is my go-to framework of choice, in a "If all you have is React/Redux then every problem looks like a nail" sort of way. I am open to reasonable proposals to adopt another framework if a sufficient case can be made.
There is no server-side component. An early experiment of this tried a server-side component as its own chat server, but this seems like overkill and you would never get people to use it, which is why we use Slack transcripts now. Once we're only parsing existing logs, this can all be done client-side, which means (a) hosting is free (b) we don't need to worry about keeping private information safe over the wire or on the back-end side. Also, since the view will eventually be in-browser, this simplies the user interaction dramatically, as we won't require users to download, install, or run any command lines things to check it out. All you need is access to the exports from a Slack channel.
User flow
It basically looks like this:
1. Add Slack logs.
Slack logs can be obtained by any owner of a Slack channel (see issue #2). At minimum, one would need two files: (1) the transcript itself (a JSON file), and (2) the
users.jsonfile, which contains data about all the users in that channel. Putting these two together is sufficient to generate a screenplay.Transcripts are broken up by channel, then by day. We may need to think through how to handle screenplays that span multiple days.
The users file contains information about the name of the user. It doesn't contain other things that screenplays usually need to know like gender or physical description. We may need to think through how to add or infer that information.
The formats are unique enough that Scriptchat should be able to understand what data is being added. There should only need to be one "add a file" button (or drag-drop onto the window to add) -- you wouldn't need separate buttons for each type. Scriptchat may either decide to maintain a collection of different files or just overwrite the previous one when a new one is added.
Providing helpful errors is a good idea at this stage:
Text files may be Fountain based, in which case we pass it straight through to Fountain for display, since we get it "for free." This makes it easy for an exported script file to be loaded through the same interface later.
2. Get a script.
Once the two Slack files are uploaded, processing happens. This step is entirely done in code. Data in, data out. This pipeline is good for anyone who knows JavaScript but doesn't want to meddle with the browser part of things. It is especially good if you like functional programming and writing tests. This is our "secret sauce," as the quality of this pipeline influences the quality of the screenplay.
Roughly, these things need to happen.
users.json:idproperty; this could get expensive. If we rewrite the data structure so that theidis a key lookup, matching users to the transcript gets easier to do.CAPITAL LETTERSand full names (if we have it) when a character is first introduced in any action. Use just the first name when spoken by another character.Also, title page
There is a special part of the Fountain syntax reserved for the title page. The view is essentially taken care of, but to populate it, there is a set of key-value pairs that make it up. It's possible that in the parsing stage we can infer some of these, but it's also easier on the user to expose the UI for it so that it can be customized.
Here is a list of all the supported title page keys: https://github.com/mattdaly/Fountain.js/blob/master/fountain.js#L225-L234
We'll incorporate that into the UI. Speaking of:
2b. Possibly tweak it a little.
In addition to editing title page properties, it's possible that the script itself might need some tweaking. Earlier, I mentioned that it would not be possible to obtain gender pronouns from Slack. We may need to provide UI after the fact to add those to users, plus other identifying information or characteristics that would enliven the script.
This step is extremely open to interpretation. Also, we may not know exactly what tweakable knobs we need to present until doing a number of test runs to figure out what really needs some human intervention.
3. Export / save it somewhere.
Once you have a script, it should go somewhere. Exporting as Fountain format is a must, since it can be read by other screenplay writing tools like Final Draft. Exporting as PDF is just as important, because that's easier to send around. Generating either of these can be done by using the token structure we have and writing that to a file.
And that's Scriptchat! In a future phase, we put on an actual production!