Estimate computation power that will be needed in the development

[MelikeSila]

Parent: #1

TODO fill the sub-tasks

[farukcankaya]

We calculated how much time the computation takes while we are developing the first recommendation software. It seems that currently, we do not need extra computation power. The most power-consuming part which is 'Data pre-processing' was be able to done on our computers. Here are some metrics:

• PDF Extraction (for the whole dataset)
  • Tika: 1h 24min 43s
  • PyPDF2: 3h 30min 45s
  • Pdfplumber: cannot be completed in 8 hours so we canceled it.
• Data preprocessing
  • Reading whole extracted text data from excel: 49.4 s
  • Concatenate the first 3 pages and cleaning: 1min 29s
  • Lemmatization: 5.41 s
  • Removing stop words: 7.35 s
  • Stemming: 5min 39s
• Topic Modeling
  • Vectorizing words(TF-IDF): 13.1 s
  • Applying SVD: 1.79 s
  • Applying NMF: 13.9 s
  • Vectorizing bag of words: 41 s
  • Applying LDA: 7min 33s
• Finding similarity
  • Finding similarity for a given input: 1.44 s

All execution time and code blocks could be found here: https://github.com/TUM-IDP-WS-20/nlp-examples/blob/feature/first-recommendation-example/Milestone_1.ipynb