architecture/api for "similar stories" subtopic definition

[rahulbot]

We are almost ready to create a new subtopic creation technique (a new "focal technique") - codename similar stories. This lets you upload a small training set of manually coded articles to train a model to find similar stories in your topic. The end result of this user flow will be a model lets us tag each story as similar to the training set of not. This is implemented as a naive bayes classifier using TF-IDF as the feature (sklearn); thresholded at 0.5 to decide if a story is "similar" or not (thx @ColCarroll).

Current Out-Of-Band Support

With @beckybell13 we've done multiple rounds of iteration using Rebekah's "Right to be Forgotten" topic. Our current out-of-band solution is a manual process that accepts a CSV of coded articles, generates a model, tags stories, and then creates boolean subtopics referring to those tags. For example, if you are looking to identify articles using a "toxic masculinity" framing in a topic about "sexual harassment", this process looks like:

1. you create a CSV of ~50 story URLS with a boolean column indicating T/F for stories about "toxic masculinity" (these stories can be outside of the topic); half True and half False
2. you email us that file
3. we run a script that generates a model and a CSV to review with all the stories in the Topic labelled T or F
4. we email you back that CSV to review for spot-checking (including recall/precision numbers)
5. you approve/reject the CSV after checking a few random stories
6. we run a script that creates a "Classifications" tag-set with two tags ("toxic masculinity" and "NOT toxic masculinity")
7. we create two boolean query sub-topics using the UI, using tags_id_stories to filter for stories with those each of those two tags This lets you browse the standard Topic Manager interface to see the results of the model and analyze them like everything else.

Integration Requirements

• We need to show the user a preview of results in the UI flow for subtopic definition (ie. so they can do validation against 30 random stories). This makes me inclined to generate the model on front-end server because this has to happen in pseudo-realtime (ie. a few seconds or less).
• The slowest part of this process right now is fetching the full text of the stories in MC.
• The back-end needs the model each time a snapshot is generated so that it can classify any new stories in that snapshot.
• The tag-base solution described above is not a strict requirement; just a short-term solution hack we came up with to make it usable for folks now to poke at.
• @kma2 is working on the UI flow now for definition; will share when I have that to help describe how this will feel

I want to start a conversation about implementation design to drive scheduling this on the roadmap. In the shorter-term, I thought perhaps it was worth considering this project while thinking about how to store word2vec models in #225, because this needs to store models too.

[rahulbot]

The UI for this is ready on a branch in the front-end. We decided to generate a model for user validation inline on the front-end with a small sample of stories (so it is fast enough that they can validate it right after uploading the training data). The key requirements on the back end are:

• a new "similar stories" focal technique, which takes as input either:
  • a list of story ids and a boolean classification column indicating if the story matched the desired classification or not (ie. the training data that the user uploaded);
  • or takes as input the serialized (pickled) model we built on the front end already (based on the training data they uploaded)
• whenever a snapshot is being created the back-end should:
  • build the model based on the training data if it needs to
  • generate a new subtopic set with two subtopics
  • focal_set names: the name the user provided
  • 2 foci in that focal_set, named "matching SUBTOPIC_NAME" and "not matching SUBTOPIC_NAME"
  • run the model associated with the focal_def against all the stories, associated each story with either foci created based on whether the result of the model (thresholded at 50%)

Does that make sense? Should we build the model on the front-end or back-end? (@beckybell13 can you link to the code that build the models so we can see an example?)

[beckybell13]

Here's the script I have that builds the model from a csv of training data, pickles the model, and optionally tags the topic stories and outputs a csv of the results.

You can see what we currently have on the front-end branch here.

[pypt]

Some notes of mine:

• Fetching full text of stories and testing it against the model will be relatively slow for us too. Boolean query subtopic (focal) technique is fast because we query for the matching stories against an indexed database, so creating subtopics (foci) and whatnot is more or less instant, but to compile the model-based subtopics, we'd have to fetch every story individually and test it against the model. My point here is that it's going to be a slow process independently from whether we do it on backend or frontend.

• I'd consider scrapping the "you email us X, we email you Y back" part because it's a manual process prone to delays, "you can't do it yourself, write us a nice letter" is a true user engagement killer, and we're half-targeting media makers who probably prefer to be kept away from CSVs, XMLs and JSONs of the world. Maybe users could create the models (and the subsequent subtopics-foci) themselves and simply delete the ones that don't work out for them, thus arriving at their perfect model by basic trial and error?

As for the architecture, I think it would be tremendously useful to decide whether the "similar stories" implementation will be exclusively a backend's or frontend's problem. The proposed "mixed" design in which the frontend generates a model and stores it on the backend in some cases and expects the backend to generate the model using the same code (shared how? Git submodule? Python package?) in some other cases just seems too brittle to me, breaks the separation of concerns and might lead to weird project synchronization issues. I'd suggest that we decide on whether:

• Generating and storing models will be done exclusively on the backend. The frontend won't generate, store or upload the models, instead relying on the backend to do everything via API calls (even for sample models for the immediate preview to the user).

• Generating and storing models will be done exclusively on the frontend. The frontend generates, stores the models and makes other decisions on them while backend is oblivious to any kind of models -- it just creates subtopics-foci based on a discrete list of stories_id as determined by the frontend (which, in turn, uses its generated model to come up with a list of stories_id to go into a subtopic-foci).

Sample user stories for both approaches:

User wants to create a new similarity model.

• Backend approach:
  1. Frontend fetches X random stories_id for sample model generation.
  2. User does the True / False supervision of the sample of the stories.
  3. Frontend submits stories_id and supervision results back to the backend; backend quickly generates a sample model using the parameter stories and tests it against a yet another sample of stories, returning the picked stories back to the frontend for user preview (model generation with 50 or so stories should be pretty quick).
  4. When user saves the model, frontend posts the final set of stories_id and supervision results to the backend; backend generates and stores a final model.
  5. Backend iterates over all the stories in a topic (?), testing them against the model (that's stored on the backend) and tagging them accordingly.
  6. Backend creates two new "boolean query" subtopics-foci for both the "positive" and "negative" stories.
• Frontend aproach:
  1. Frontend fetches X random stories with full text for sample model generation.
  2. User does the True / False supervision of the sample of the stories.
  3. Frontend generates a sample model using the supervised stories, fetches another sample of random stories with full text from backend and filters this sample using the model for user preview.
  4. When user saves the model, frontend stores the generated model on the frontend's side.
  5. Frontend iterates over all the stories in a topic (by fetching full text from the backend), testing them against the model that's stored on the frontend and tagging them accordingly using API calls.
  6. Frontend creates two new "boolean query" subtopics-foci for both the "positive" and "negative" stories using an API call.

Snapshot gets generated and the new stories in it need to get run against the model.

• Backend approach:
  • Backend tags new stories using a locally stored model by fetching story full text content and testing it against the model.
• Frontend approach:
  • Frontend learns about the newly generated snapshot somehow (Cron job?)
  • Frontend fetches a list of untagged stories in a snapshot.
  • Frontend fetches full text of untagged stories, runs the text against the model and tags individual stories accordingly via API calls.

[rahulbot]

Thanks for thinking those options through. I think we have to use the back-end approach.

With that backend solution:

• I think we need to save the model with the snapshot to have a good reproducibility story (ie. we need to show a "download the model" link on the "manage subtopics page")
• Saving the model I think requires us to create a new focal_technique, rather than overloading this onto boolean query
• And that means tagging stories isn't the right approach; we should instead assign them to foci_id like we do for the other foci

If we are all agreed, I think the next step is to spec the API changes needed.

[rahulbot]

Ping! This is overdue. Please turn this into an API spec this week that we can review together.

[pypt]

Some initial questions:

• what's a focal set definition?
• is there a way to fetch a list of stories from a topic snapshot, i.e. how do you do topics/<topics_id>/snapshots/<snapshots_id>/stories/list?

[pypt]

Also:

• Why would you need to download a model given that we've decided on the "backend approach"? Doing the task only on the back/frontend would save us from having to keep the code that generates / uses the model in sync between the -ends.

[rahulbot]

A topic has focal_set_definitions, each of which has many focus_definitions in it. When a new snapshot is generated, these get turned into focal_sets and foci, which are associated with that specific snapshot.

To list all the stories in a snapshot, you pass in a snapshots_id param to the call, like topics/<topics_id>/stories/list?snapshots_id=<snapshots_id>.

[rahulbot]

With the "backend" solution we agree on, you are correct that we don't need to allow the model to be downloaded for any system reason. I think we need to allow it to be downloaded to have a good reproducibility/replicability story. I want to be able to tell a user that they can download and use the model and use it independently.

[pypt]

A topic has focal_set_definitions, each of which has many focus_definitions in it. When a new snapshot is generated, these get turned into focal_sets and foci, which are associated with that specific snapshot.

Yes, I get that from the database table structure, but what does it all mean? :)

With the "backend" solution we agree on, you are correct that we don't need to allow the model to be downloaded for any system reason. I think we need to allow it to be downloaded to have a good reproducibility/replicability story. I want to be able to tell a user that they can download and use the model and use it independently.

I can add the call which would return a .zip of the model (it consists of two files apparently), but that might not be too useful for the user as the model appears to be readable only by a specific version of sklearn, also we'll need to maintain a Python code sample for the user to load the model (which might go out of sync soon), etc.

[hroberts]

The focus and focal_set refer to the thing actually created by the snapshot. The focus and the focal_set will never change because they are part of that static snapshot. This is necessary because we don't want any of the results in a snapshot to change over time.

But we often make a series of snapshots over time, so we need a stable definition of which focal_set and foci to create for any new snapshot. So the focus_definition and focal_set_definition objects exist to tell the system which foci and focal_sets to create for any new snapshot for a given topic. They exist as separate objects from the snapshot because even though in theory they might change between every snapshot, in practice they are usually pretty static, so it would be a pain to require the user to recreate them every time.​

From a user point of view, the focus_definition and focal_set_definition is what you are editing when you edit the subtopics for a given topic. The foci and focal_sets are created using those definitions at the time of the actual snapshot.

You can specify a snapshot with the snapshots_id= parameter:

https://api.mediacloud.org/api/v2/topics/2306/stories/list?snapshots_id=2296

Note that you are really always querying a specific timespans_id. If you specify the snapshots_id, the api just selects the overall, no subtopic timespan for you. If you specify neither the timespan nor the snapshot, the system chooses the overall, no subtopic timespan from the latest snapshot.

[rahulbot]

I think the process should kind of work like this (based on your outline earlier):

• User decides to create a new subtopic in Topic Mapper UI
  • Front-end generates CSV of X sample stories for user to hand-code
• User uploads CSV of ~100 sample stories (training set) with a boolean judgement column they filled in
  • Front-end posts training set stories_id and judgement list to back-end
  • Back-end pulls story text for just the training set and trains a quick model
  • Back-end runs trial model on 30 more stories that were not part of uploaded set (validation set)
  • Back-end returns judgements on validation set, model precision, model recall, and model top positive/negative words to front-end
• User users front-end to confirm model works well, names the subtopic set and saves it
  • Front-end sends training set and validation set results as stories_id/judgement pairs to server, with name for new subtopic set
  • Back-end creates new focal_set_definition of type "similar-stories" with name from user (ie. "pro-Brexit"), creates two focus_definitions that are children of that focal_set_definition - one for "is " (ie. "is pro-Brexit"), one for "is not " (ie. "is not pro-Brexit")
  • Back-end creates final model and saves training data with it (so user can later download either)

When the user later decided to create a new snapshot:

• Back-end reads "similar-stories" focal_set_definition, creates focal_set and 2 foci (associated with model and training data)
• Back-end iterates over text of every story and runs it through the model. If model results > 0.5 then story goes into "is X" focus, if < 0.5 then the story goes into the "is not X" focus

It is up to you whether to use the tagging architecture and boolean foci for this or not; I don't know how stories are generally associated with foci.

[pypt]

Back-end creates focal_definition of type "similar-stories" with name from user (ie. "pro-Brexit")

@rahulbot, what's a focal definition? Is it a focus definition or focal set definition?

[rahulbot]

Good catch. That should say "back end creates focal_set_definition of type "similar stories"". I'll update it now.

[pypt]

Sorry again for the long delay, this (foc(us|al|i)|subtopic)( set)?( definition)? stuff is very hard to approach by the uninitiated (I'd argue that we need to simplify a little by at least renaming a bunch of stuff).

Here's my API draft, tell me what you think:

User wants to preview a new similarity model

1. Frontend two sets of random stories for sample model generation using a pre-existing topics/<topics_id>/stories/list?sort=random call. One set is to be used for training, another one for evaluation.
2. User comes up with True / False judgements of the training sample of the stories on the frontend.
3. Frontend posts training and evaluation sets of stories to topics/<topics_id>/similarity_models/preview call.
4. Backend quickly trains a sample model using the training story set, evaluates the evaluation story set using the sample model, and returns back the judgements back to the frontend.
5. Frontend fetches another set of stories (see step 1) and comes up with a new improved training set with the help of the user.
6. Rinse and repeat until the user is happy with the training set of stories.

Notes:

• Not sure if backend can return precision / recall metrics because only the frontend (i.e. user) knows which stories are the "right" ones to go into the model.
• Haven't found a way to extract "model top positive/negative words" from Becky's code.

User wants to store a new similarity model

1. Frontend posts a training set of stories to topics/<topics_id>/similarity_models/create call
2. Model gets stored on the backend, API returns created model's description with similarity_models_id field in it.
3. Frontend creates a new focal set definition using topics/<topics_id>/focal_set_definitions/create call.
   • Similar Stories "focal technique" is to be used to create focal set definition for similar stories matching.
4. Frontend creates one or more focus definitions using topics/<topics_id>/focus_definitions/create call.
   • A single query argument which supported only Boolean Query "focal technique" has been replaced by focal technique-dependent args argument.
   • Focus definition can be made to "negate" the similar stories model by setting negate to true.

[rahulbot]

I see your approach is to split the model generation into its own process/endpoints, and then to connect the model to the focal_set_def once it is ready. That seems reasonable. A few notes/questions:

• shouldn't the required roles for both similarity_models/preview and similarity_models/create be topic-admin?
• similarity_models/preview should return precision and recall of the training set following Becky's example here - the backend has the "right" judgement for all the training set so it can do this I think. It should also return the top words associated with pos/neg result code here demonstrates how.
• Does calling similarity_models/create actually generate the model (ie. take a few seconds), or just queue the model up to be generated once the snapshot is run?
• How do I download a model? Does the similarity/list call include a url to download the model from or something?
• I think the similarity models need to include the user_id of the person that trained them (in the results of similarity_models/list at least)
• I like your generic args property on focus_definitions/create & focus_definitions/update; it shouldn't be hard for us to refactor the existing calls to make it work that way

[pypt]

Thanks for the feedback!

• shouldn't the required roles for both similarity_models/preview and similarity_models/create be topic-admin?

I've made "/preview", "/list", "/download_model" and "/download_vectorizer" require "tm-readonly" role, and "/create" require "tm" role.

• similarity_models/preview should return precision and recall of the training set following Becky's example here - the backend has the "right" judgement for all the training set so it can do this I think. It should also return the top words associated with pos/neg result code here demonstrates how.

Thanks, precision and recall weren't present in the original sample. Added "precision" and "recall" floats in "/preview", "/create" and "/list" responses.

• Does calling similarity_models/create actually generate the model (ie. take a few seconds), or just queue the model up to be generated once the snapshot is run?

"/create", as well as "/preview", are supposed to generate the model right away.

• How do I download a model? Does the similarity/list call include a url to download the model from or something?

Given that the model consists of two files - model data and vectorizer data, I've added two API calls to download both:

• https://github.com/berkmancenter/mediacloud/blob/similar_stories/doc/api_2_0_spec/topics_api_2_0_spec.md#topicstopics_idsimilarity_modelssimilarity_models_iddownload_model-get
• https://github.com/berkmancenter/mediacloud/blob/similar_stories/doc/api_2_0_spec/topics_api_2_0_spec.md#topicstopics_idsimilarity_modelssimilarity_models_iddownload_vectorizer-get

• I think the similarity models need to include the user_id of the person that trained them (in the results of similarity_models/list at least)

I've added "auth_users_id" field to "/create" and "/list".

[rahulbot]

That all sounds great. Two further reactions:

• Regarding permissions - what I was trying to say was that this should respect the topic-level permissions system. Ie. I should only be able to call this stuff as a user for topics that I have admin-level permissions for. If I don't have any permissions for a specific topic I shouldn't be able to call these. This is different than the overall system-level roles.
• Regarding are model author info - right now I don't have a way to get actual name of a person based on their auth_users_id. I suppose that'll change if we do what I suggested for the user management API (https://github.com/berkmancenter/mediacloud/issues/404#issuecomment-403606308), but failing that it'd be super helpful to include the user's actual name in the /create and /list response so I have something useful to show in the UI.

[pypt]

Thanks on the clarification regarding the permissions. I've made the spec to require either write (for /create) or read (for the rest of the API calls) permissions to the specific topic.

Also, I've replaced auth_users_id with owner which contains full_name parameter, among others.

[rahulbot]

API agreed in. Moved implementation to separate issue.