Data(preprocessing) - Githubissues

dfsbora / latex-math-model

0 stars 0 forks source link

Closed ricobruland closed 4 months ago

ricobruland commented 4 months ago

Source more data and implement preprocessing.

dfsbora commented 4 months ago

Basic preprocessing generates a single file data.tex clean of comments, empty lines, chapter formatting, and filtered content.

dfsbora commented 4 months ago

Uses eos tag as a separator for meaningful blocks (e.g. keeps a proof environment as a whole thing).
Updated data.tex and also created a smaller data from two short chapters for quicker tests (data_small.tex)
Evaluated size of a meaningful blocks (Mean: 162; Median: 123)

dfsbora commented 4 months ago

Proposal to use plain text was rejected.