buffer

[mikekestemont]

I sketched a first version of the settings objects and the buffer thing. Everything gets called from train.py.

• Currently, the shuffling for the buffering is only done at the file level. This is OK for now, right?
• Please check the internal structure of the buffer (a tuple of sentence dicts with token, pos and lemma). I did this as a dictionary at the sentence level, because that will be easier later when people decide to leave out certain categories. OK?
• I am assuming the hard case where users do not have sentence boundaries and we simply take fixed-length slices from a text.
• Next up is the featurization of the buffers (with a separate loop for fitting); I'll base myself on the objects in seqmod, but probably it's better to re-include them here separately to avoid dependency issues. OK?

[emanjavacas]

My suggestions:

• Since we will probably expect different types of input formats, this could be abstracted in a reader class that yields line by line examples (perhaps with an option to do this in windows for the case where we can to look at the previous, or next sequence). The dataset will then take a reader as input.
• Instead of a dict, you probably want to use a namedtuple, which is immutable, so you don't need the deepcopy operation.
• I think it's better to not depend on seqmod at all indeed.

BTW. I just implemented my suggestions in code, cause it was simpler than actually describe them. I've pushed it to brach sketch/data

[emanjavacas]

I've added some missing parts. Extra things to discuss at this level are:

• how to deal with/what to do when we don't have data for a given modality
• build support for windowing over sentences (easy to add, but the question is do we need this? probably we can experiment with it at some point).
• provided lemma char-level processing on demand