[Project] Possible NLP applications for repository

[monotasker]

• get text from uploaded files (pdf, docx, pptx)
  • research available python libraries for text extraction
• prepare text
  • clean
  • separate languages
  • lemmatize/stem
  • POS tagging
  • embeddings
  • named entity recognition
• kinds of analysis
  • word collocations
  • topic modelling (unsupervized? supervized?)
  • network analysis
  • sentiment analysis
• how can subject headings on deposits be used as kind of classification
  • FAST subject headings (LOC headings)
  • has anyone developed tools for LOC headings?
• how can other metadata be used for analysis?
• how can we do auto summarization? free LLMs?

[cassandralem-msu]

Another possible thing to do: create network graph to improve recommendation systems

[cassandralem-msu]

Python library for .pdf text extraction: pypdf Article describing 3 different Python libraries for .docx text extraction Python library for .pptx text extraction: python-pptx

[koutiany]

I used this one before: https://pypi.org/project/pytesseract/ Amazing support on multilanguage options: https://tesseract-ocr.github.io/tessdoc/Data-Files-in-different-versions.html

[koutiany]

embeddings.pdf

The document about embeddings. I mentioned collocation and embeddings when we have clean corpora. Collocations focus on the frequent co-occurrence of words and their specific combinations, while word embeddings aim to represent words in a continuous, semantic space to capture their meanings and relationships

[koutiany]

https://huggingface.co/docs/transformers/index this is where I get some of my knowledge from. :) Wish i could spend more time here.

[koutiany]

Python lib summary for text extraction see here