Having an initial look at data

[vaidap]

Advice from Alexander:

• Not worrying too much about getting perfectly cleaned up dataset. Accept noise
• Writing scripts/functions to pull aspects of data we are interested in eg. headers, using regular expressions
• Using spacy (recommended), or nltk with python to analyse data
• For Natural Language Processing: use plain text, for spatial information: structured data so XML (built-in XML package in Python), there's XML readers

The next steps would probably be:

• [ ] Read about Encyclopaedia Britannica & its context
• [ ] Have a look at the different versions of the data: what we want do do with it
• [ ] Think about how to clean up the data (first try operations, then see if we need to clean)
• [ ] Think about individual assignment
• [ ] Have a think about possible group assignment?
• [ ] Get in touch with the data holder
• [ ] Talk about time management & timeline
• [ ] Look at case studies of similar projects
• [ ] Check handbook evaluations of project

[vaidap]

Advice from Sarah

• We can talk to librarians that have domain knowledge about this data
• She recommended understanding the data & thinking about a specific aspect to work on/a specific subset
• https://programminghistorian.org/
• Thinking of an audience: public, researchers
• Topic modelling?

[vaidap]

s/f letters misspellings

• unique words -> compare with dictionary, and manually write equivalences & map
• if people use modern word: do a replace & search for old word possibility & similarities
• replace for all words, s & f with underscores, and replace modern dictionary with _

idea

• new meaning of old word, searching in wikipedia, comparing: how knowledge evolves
• would need additional data: new data (scraping web data), wikipedia probs have API
• word embeddings: word vectors. train on old data/pretrained, and compare between old/modern encyclopedia. gensim

1) question: measure change 2) method: how do we measure change? 3) what do we learn from the data from this method? "giraffe" is stable word can select subset of words, and clean just those.

• measuring changes of words across editions: how would you compare it with itself?

example vizs:

https://fathom.info/traces/ (tableau?) -> shows additions/deletions, and growth http://notabilia.net/ https://www.c82.net/work/?id=347

size of xml documents: headers -> look at length of words in section, index across editions and compare what exists across editions. build dictionary of keys, values (name, length, content): compare dictionary keys. check if word is mostly capitalised rather than full capitalised.

• try extracting headers, sections, try to remove noise
• on slack: give alexander idea on what we're doing

[vaidap]

I have an exploratory.py in the github that loads a text file into an ipython notebook!