search

dougpuett / webapp

Code for my website
Eclipse Public License 1.0
0 stars 0 forks source link

Establish note taking priority set #38

Open dougpuett opened 9 years ago

dougpuett commented 9 years ago

If I have a document, what are the relevant notes that should be included in the message stream? Possibilities: -explicit notes linked to text body -notes linked to shown notes (collapsed?) -notes on texts linked to current text (graph search problem) -topic modeling (how to do this?!)

each note should be created with a context that is recorded and understood along with the message text. this greatly increases the text data the note contains and allows topic matching as well as graph operations to be performed in a straight-forward manner.

A long text may be represented thusly: the piece itself is a node, each chapter is a node connected to the piece as well as the previous and subsequent chapters. each paragraph fallows similarly and then each sentence and/or line. In addition, there exists the ability to use topics or other textual features as as nodes. a single long text may be considered the collection, with each paragraph or set number of lines the document. LDA can be run on the collection and weights are then established between each document and topic. Each note then has a weight over various topic features that can be used as an input in the scoring mechanism.

dougpuett commented 9 years ago

I think this approach would require a graph based database. I'd likely use neo4j, although there are obviously some alternatives, including my own DSL on top of a more traditional database (probably mongo). this is likely to be too difficult primarily because of the indexing problem, which I have no interest in.

dougpuett commented 9 years ago

a good understanding of graph weights enables a single solution instead of trying to balance and federate a bunch of different heuristics

dougpuett commented 9 years ago

The question then becomes: how to efficiently explore and weight a large graph?

dougpuett commented 9 years ago

I am more interested in path discovery than node discovery. Is this supported by neo4j?

dougpuett commented 9 years ago

this appears possible in neo4j, but isn't the most typical operation, especially when you consider how important edge weights are

dougpuett commented 9 years ago

steps in document acquisition: -make sure you have the correct text -create subdivisions and linkings -create a topic model for the document -score against existing LDA model -create new LDA model