longer markdown cells in the beginning/end

[Isabel-Gan]

will not catch the case where there may only be a few markdown cells, all placed at the beginning or the end

this would fit in the definition of the feature, but the code will not catch it as it calculates an average length across all markdown cells

[Isabel-Gan]

will not catch the case where there is a constant length of markdown cells (see nb id 165313, incorrectly says "true")

[Isabel-Gan]

define a "longer" markdown cell to have a length of at least 5 lines

[Isabel-Gan]

issues:

• 492523, it's long but counting the numbers of strings in source will not suffice
• 1019165, same as above

[Isabel-Gan]

possible solution: count md cells by character instead???

[Isabel-Gan]

the above issues are resolved, but new issues (false positives):

• 464706
• 333748
• 578489
• 165313

the issue is that it's counting stuff like html tags in the character count, when those don't actually show up in the markdown cell

[Isabel-Gan]

maybe going by line was the better way to do it? or filter the actual text out from those lines somehow?

experimenting in https://github.com/Isabel-Gan/quantifying-notebook-features/tree/quantifiying-markdown-length

[Isabel-Gan]

possible fix: https://stackoverflow.com/questions/328356/extracting-text-from-html-file-using-python

[Isabel-Gan]

"fixed" by https://github.com/Isabel-Gan/quantifying-notebook-features/pull/11, but still results in some of the false positives above. upon re-inspection of the notebook, the first markdown cell was actually one line longer than the rest, and the script is correct

[Isabel-Gan]

found a lot of false positives in the actual run, need to change how this is measured

[Isabel-Gan]

https://docs.google.com/document/d/1lucE05_8DUCuKVK4yTYqhlVnMDt4r91bokttCQM01mk/edit?usp=sharing

[Isabel-Gan]

results from discussion with Shurui:

• should count the number of words instead of characters
• count links, equations as one word
• trim the text before calculating the length

[Isabel-Gan]

fixed by https://github.com/Isabel-Gan/quantifying-notebook-features/commit/c43c7c937cac26ca01277f49edd72a48953c9b05 and https://github.com/Isabel-Gan/quantifying-notebook-features/commit/44433bc276111588874cfe4e8a0f1b153c3678c1

[Isabel-Gan]

last round of fixes:

• don't quantify this metric on notebooks with <10 cells, 502567: https://github.com/aprilnovak/coupling-moose/blob/26999ebb4cda8504510d545ec441240ac8495d71/modules/phase_field/examples/anisotropic_transport/README.ipynb
• lengths of markdown cell groups rather than individual markdown cells, 958614: https://github.com/ArtemKupriyanov/MOOC-COURSES/blob/21efc69b5061b96efdb9b59d6a4a9961436c1a81/ML-MIPT-Yandex-spec/2.supervised_learning/Homeworks/linear%20regression%20and%20python%20libary%20fo%20AD/peer_review_linreg_height_weight%20(1).ipynb

[Isabel-Gan]

fixed by https://github.com/Isabel-Gan/quantifying-notebook-features/commit/9baab0d5c327b01eb9d888f833e732b9e74a60a2 and https://github.com/Isabel-Gan/quantifying-notebook-features/commit/c150528d0b78ffed6ca49ecabcca4685b10c69cf