Notion of Raw cards, with associated workflow, to make accumulating raw ideas easier

[jkomoros]

There's the 80-page catalog of rough content in the "staging" doc. The idea originally was that everything would be public, and concepts would "bake" from "Random Thoughts" upwards, accumulating rough ideas like that.

But in practice that's not what happens. Perhaps because Random Thoughts things do get tweeted? Or perhaps because the random parking lot for an idea is the artefact that will later be polished and shared? And also because a card has a finite length. There's a desire to make the card stand on its own and be full... but then that takes up all of the space and then later when you realize more ideas to connect you can't and there's no room.

Ideas are shifting as they accrete and distill. Ideally there would be a subterranean staging where you literally just free associate into a card-like thing that will never actually be cleaned up into a real card. And then the system processes n-grams or something and finds concepts to pop out, into NEW cards that link to the staging material. The staging material still exists, it's still in the web, but it's subterranean.

In general, the idea of finding interesting n-grams and allowing searching cards that way is interesting.

Maybe have a type of card that's called 'staging' or something. No title. Just a big text box. You also can't ever 'publish' a stagingn card. So the author can see the staging content, but no one else can, and there's no feeling that that content will ever be exposed.

And then have a flow to take a staging card, copy it, retain a link to the card it staged from, and then not have to worry about "losing" anything. The content just accretes.

And there would be a quick flow for editors to 'Collect Idea' that would automatically take you to a new card in the staging section. Staging tab would never be visible to anyone other than admins (or people who are allowed to create cards).

(The permission to create staging cards might be more common than the permission to create content cards)

So two ideas here: a staging card type and associated flows, and auto-n-grams that are kind of like tags, but automatic, and helps an editor or writer naturally see other cards that are related

[jkomoros]

Cards should have a references field, which is basically cards that this card links to but that aren't explicitly in the body. The extractCardLinks would also include those, so card.links would have them. ideally this would use exactly the format of whatever citations are, and be part of the same machinery (so maybe a map of card-id to true or string). That's how a card that's based on a raw card would show it's connected, it would automatically have that field set. Obviously those would be elided if the cards they point to aren't published, which raw cards never are.

Create Content Card button when editing a Raw card would create a new card, paste in that card's body, leave an empty title, and have a reference back to the raw card.

Similar cards--show many more while editing.

Similar cards should look into doing terms based on n-grams as well, the finger-prints might get higher quality, since some key terms are actually ngrams. For example, we could look over the whole corpus to find n-grams that appear to coccur a lot and then use those n-grams as the actual terms to fingerprint over.

Make a "Display Semantic Fingerprint" that basically takes the fingerprint for the card, and re-expands the stemmed words into real words, based on looking at the card and figuring out which stems match the word from that card. These Display SEmantic Fingerprints (perhaps with a date preprended) would be used in place of explicit titles for Raw cards, making them easier to grok when they show up in similar cards.

A way to view all similar cards for the current card, sorted by descending similarness, and with cards that are already linked grayed out? (This is captured in another issue somewhere about allowing similarCards to be done as a collection descriptor)

Raw cards shouldn't have a title, just one big text box, with text styling that is smaller than other cards so more content and ideas can fit.

[jkomoros]

Ideally raw cards would also be able to be sorted based on which ones a) aren't referenced yet by other cards ("undeveloped"), and b) have the fingerprint that overlaps with the most other undone raw cards (cocnepts that seem interesting)

Are raw cards in a special 'Raw Cards' section that's generally hidden from readers, or are they all orphaned? Is there a way to get literally all cards as a base set, including orpahned, so you have something like /everything/type-content/. (Isn't there another issue about how orphaned is a hard filter to do because 'all' isn't literally 'all'?)

[jkomoros]

The title of raw cards should be computed once when saved, and not derived live.

It should look like > 7/23/20 Trust Momentum Zig-or-zag...

Where the words are the top, de-stemmed fingerprint words, and the> is some kind of bullet to make them stand out in lists of card-link that have normal content cards in them

[jkomoros]

Should raw cards be orphaned, and accessible via /everything/type-raw/ or should they be mixed into normal sections, and typically in a 'Staging' section?

Pros of orphaned:

• Raw cards don't accidentally show up in other sections
• No need to do special logic to hide the raw section
• Handling orphaned cards is work that should be done anyway
• New cards don't need to point to a different section to be put in, just a bool of "should new cards be orphaned"

Pros of section:

• There's a clear ordering of raw cards that can be changed
• Raw cards can show up in things like recent lists with no extra work (although often you specifically DON'T want that)

In either case, we'll need to do some special behavior for new cards: new content cards go in the lowest normal section, and new raw cards go either in the staging section or orphaned.

... I also think that maybe it's better to shift to a model where every card--even orphaned ones--have an intrinsic order, so even orphaned cards can be ordered. Issue #199

[jkomoros]

The action to take a raw card and start working on it, while keeping a refefrence to the original, is a "fork" action. Also useful in e.g. #328. Would there need to be a special field for which card it was originally forked from, in addition to a generic reference to it?

[jkomoros]

OK, given that we'll do #199 at some point, we'll be able to reorder even orphaned cards, so raw cards should just be orphaned.

Order of operations:

• [x] Allow orphaned cards to show up #358
• [x] Add a working-notes card type
• [x] Create a filter for working-notes and content
• [x] Create an action-creator to create a new working notes card (alllow createCard to take a working-notes card type ... and document what those opts are, by the way)
• [x] Add a new note card button in collection if the user has edit cards access)
• [x] Fix styling of working-note cards. The text should be much smaller. (Make sure the longest block of notes I have in cards staging will fit)
• [x] The canonical location for a working-notes card shoudl be everything/working-notes/CARD_ID, so that it goes there automatically when you create it
• [x] Is there a bug where when you're viewing an orphaned collection and you add something, the other cards get styled like they'll get removed in next collection commit?
• [x] When you create a new working notes card it doesn't navigate, no matter what
• [x] Clean up pasted content from staging more (e.g. remove 'dir=ltr' and 'role=presentation' and p tags shouldn't be allowed inside of lis)
• [x] Title extractor for the raw card type
• [x] Title extractor for raw card type should use a shorter date.
• [x] title extractor for raw card type should include the first bit of a semantic fingerprint
• [x] Weird circular card-fingerprint thing where the date from the title might be included in the fingerprint, because title is used in calculating it... So you get things like cards without much content include the date-string at the beginning. really the problem is that we're treating the derived title like it's a real field but it's actually just derived... . Have a DERIVED_FIELDS map by card type, and then have the extractNormalizedFields or whatever use blank values for those?
• [x] the semantic fingerprint should use the EDITED card (currently it uses the baseline card)
• [x] Fix the race conditiion where if you commit a working-notes card before the debounce has happened for extracting noramlized fields happen, it uses the thing from before the bounce. Oddly, fingerprint DOES seem to be calculated live for... tags? But for the actual body content, it's not extracted until after a short delay, for performance. When we commit editing it should for an extraction. And perhaps the suggested tags shouldn't be live extracted (which I think they're doing, based on how often the console dumps fingerprints being calculated)
• [x] Createing new working-notes card when viewing working-notes updates the URL but the card doesn't show in the collection or on screen until you refresh. updateSections has a special case to force updating the collection if there's a single section that loaded. That missing force section is why the collection doesn't update, and we can't select the new card.
• [x] Hitting the new working-notes card should create it and automatically open it for editing (shouldn't normal cards do that, too? was it just the bug immediately above that made it not do that?) (It doesn't work as you'd naively expect to open for editing because the serverTimestamps are null see https://medium.com/firebase-developers/the-secrets-of-firestore-fieldvalue-servertimestamp-revealed-29dd7a38a82b)
• [x] [dream] Cards connection in related time and links is the hidden flow of implied connection of ideas. Overall this makes the discontinuous process of notes --> cards much more continouous, unlocking content and momentum.
• [x] To back out the semantic fingerprint, the memoizedStemmer could keep track of, for each output, the count of the inputs, and then when asked for the reverse direction, return the most common. That wouldn't have to be that optimized, because it would only be done for a handful of words each time a working-ntoes card is saved. But we'd ideally want the card to use the de-stemmed variant that is most popular FOR THIS CARD. To do that, re-stem the words of the card on save, keeping track of the raw word and the stemmed variant, and then do the same calculation (but with a smaller backwards map)
• [x] semantic fingerprints should be created for working-notes cards too
• [x] Title extractor for raw card type should include de-stemmed semantic fingerprint
• [x] Show a 'Working Notes' section in header if you have any working-notes cards that you can see (hideOnEmpty:true)
• [x] The button to create working notes should go to bottom right (it's currently direclty underneath the normal add button)
• [x] working-notes card type defaults to orphaned cards
• [x] generalize it so normalized* properties, set in util/cardSetNormalizedTextProperties is done automatically driven off of configuration in card_fields.js ) (also combine with TEXT_SEARCH_PROPERTIES). (All normalized properties should be put on a normalized sub object where their keys in that are literlaly the same as their keys in the normal card. Does search just look at every single field in that normalizd object? And then setNormalizedFieldsForCard (or whatever) runs through the list of configured properties and extracts them based on a typeof the field and the config on if html, and if derived)
• [x] card links to a non-content card should be in italics
• [x] working-notes cards will warn if you try to publish (and DON'T warn if content but not published)
• [x] Card thumbnail should make it more clear it's a working note (also, badges are wrong color)
• [x] working-notes cards will warn you if you try to make non orphaned
• [ ] Card titles should be allowed to be set on working-cards (as a kind of 'proto title', just do the current behavior if not provided. Same behavior for normal cards? (where the missing title will be far less common)
• [ ] Allow starring cards or otherwise some way of showing priority to develop next. (Production uses starring but that doesn't sound right)
• [ ] Call cards personal-notes, and make it so 'viewUnpublished' permission doesn't automatically mean access to them? (change name of config item in tabs.js for working notes) (What is the way for a given user to give other readers access to their notes cards but no one else's? ... In practice there's one primary author in a card-web so it doesn't really matter)
• [ ] The date used in working notes titles should be able to be set (since the notional date of the intake is not necessarily when they happened, especialy for legacy cards) (note: a number of working-cards in production have the original date as a free-form todo)
• [x] #354 - reference to cards
• [x] #357 - fork action for cards (#360 might be relevant first)
• [x] REmove special machinery for card-type filters and use generic CARD_FILTER_CONFIGS configuration that's auto-generated
• [x] Create a 'card-type-has-body' CARD_FILTER_CONFIGS
• [x] create a selector that reutnrs the types of body-card card-types the user can currently see
• [x] The find card dialog looks up if the number of card-types is greater than 1, and if so, renders a radio button for (default / and one per card type) (actually hook up in state)
• [x] The filteredCardType is added in at the beginning of selectCollectionDescriptionForQuery if it exists
• [x] If you select a non-default cardFilter in find and then close it and then reopen, the model and the radio button selected are out of sync (update to lit-element 1.2.0 and use the live directive: https://github.com/Polymer/lit-html/issues/1084)
• [ ] try pasting in contentwith lots of extra newlines (e.g. from gmail) to see if more normalization is required for that case (#364)
• [x] Find dialog should default to searching over normal content cards, but allow the option of searching over just notes cards (if the user has view access to at least one notes card) (perhaps the rule is: any cards that have a body, which is now BODY_CARD_TYPES). Just update selectCollectionDescriptionForQuery. Make the card-type filters be added to Card filter configs automatically, and not have special machinery. also add the 'card-type-body' filter. Then have the default collection be `everything/card-type-body', and if the user has access to more than one type of card that pass that filter, then show a selector to allow default (no additional filter), or to add a card filter for either type of cards.
• [x] Allow anyone with edit access to create an orphaned raw card
• [ ] Auto toods for notes cards shoud exist--but be different. (E.g. no todo to publish) (#384)
• [ ] A way for a notes card to be shown to be 'fully tapped' or not. A fully tapped card has had all of its meaningful insights popped out into cards. Some working-notes cards only need one, some need multiple. Have 'not-tapped', 'patrially-tapped' (at least one forked reference), and 'fully-tapped' (a manual upgrade from partially-tappe) ('mined'?) (fully mined notes by default are filtered out of e.g. similar cards, etc)
• [x] There has to be some way to delete working notes cards (but you should have to check that no other cards inbound link here) (and tags don't reference it, and sections don't reference it) (and only if it's orphaned) ( now captured in #383)
• [x] Ctrl-N is equivalent ot hitting the 'craete-working-notes-card' button to hitting it (if another card isn't already open for editing). Ctrl-M (n?) should create a new card, either in this section if it's allowed, or if not, in the last section, and navigate to it and open for editing immediately. (As long as another card isn't being edited). Ctrl-Shift-M/N should do the same but for wokring notes cards
• [ ] Document working cards usage patterns in wiki
• [ ] Similar cards list should be able to filter out types of cards #317 (so there'd be a Simile Cards and Similar Working Notes Cards section)
• [ ] Allow a similar cards that is about recency of creation time (helping find the related notes cards that were created at roughly the same time)
• [ ] Fractional ordering for cards so orphaned raw cards can be reordered. #199