Scraper PoC

[VladyslavKurmaz]

Goals

• Select a set of tools/frameworks to be used during development RESTful/Unit Test/HTLM parser/JSON parser frameworks
• Define proc/cons for thread pool usage
• Create UML diagram (or java classes skeleton) for scraper service, including hierarchy of classes, helper classes etc.
• Collect ideas howto implement persistent layer - storing downloaded information
• Take into account extendibility & reusability, in other words, prepare design for future extensions
• Use dou.ua as first candidate for scraping information
• Create parsing cycle: resume -> scan -> suspend

Stages

• [x] Create maven-based Java projects under services/api folder
• [ ] RESTful framework can be Jersey, https://jersey.github.io/documentation/latest/getting-started.html, but I would recommend to check this article https://www.gajotres.net/best-available-java-restful-micro-frameworks/ and other resources before starting. Another good candidate can be Play Framework. Please check all possible alternatives and suggest a solution.
  • [ ] Include Unit Tests framework JUnit and check this article https://dzone.com/articles/10-essential-testing-tools-for-java-developers and suggest does it make any sense to select another framework

[fac30ff]

As I review the tools and frameworks for RESTful/Unit Test/HTML parser/JSON parser : Spring Boot (Spring DATA for building persistent layer with mongoDB), JUnit (for stub )/Mockito (for mock), JSoup, Jackson (JSR353).

[fac30ff]

Methods Flow:

• configureTor
• launchTor
• shutdownTor

  ---

• configureScraper
• takeLinks
• sendRequest
• persistInfo
• storeInfo

  ---

• configureParser
• readInfo
• parseInfo
• storeShortInfo

  ---

• ErrorHandlers

  ---

• configureAI
• readInfoShort/Full
• learnAI
• parseByAi
• SuggestingToUser

[fac30ff]

Persistent Layer:

1. DataDao - interface
2. siteFullDataDao - implement
3. siteShortDataDao - implement | decorator | projector

All task can be done with Spring Data.

Storing in mongoDB.

[fac30ff]

UML diagram:

link to diagram to gliffy

[fac30ff]

Issue image do no to load but reference valid; The first time I did the diagram in the service gliffy, without thinking, the next time I'll do it in draw.io.

[fac30ff]

Propose to add in .gitignore next line: for java:

Compiled class file

*.class

Log file

*.log

BlueJ files

*.ctxt

Mobile Tools for Java (J2ME)

.mtj.tmp/

Package Files

.jar .war .nar .ear .zip .tar.gz *.rar

virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml

hs_err_pid*

for maven: target/ pom.xml.tag pom.xml.releaseBackup pom.xml.versionsBackup pom.xml.next release.properties dependency-reduced-pom.xml buildNumber.properties .mvn/timing.properties .mvn/wrapper/maven-wrapper.jar

backup files: .bak .gho .ori .tmp

jetBrains IDE's:

Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm

Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839

User-specific stuff

*.iml .idea/ .idea//workspace.xml .idea//tasks.xml .idea//usage.statistics.xml .idea//dictionaries .idea/**/shelf

Sensitive or high-churn files

.idea//dataSources/ .idea//dataSources.ids .idea//dataSources.local.xml .idea//sqlDataSources.xml .idea//dynamic.xml .idea//uiDesigner.xml .idea/**/dbnavigator.xml

Gradle

.idea//gradle.xml .idea//libraries

CMake

cmake-build-*/

Mongo Explorer plugin

.idea/**/mongoSettings.xml

File-based project format

*.iws

IntelliJ

out/

mpeltonen/sbt-idea plugin

.idea_modules/

JIRA plugin

atlassian-ide-plugin.xml

Cursive Clojure plugin

.idea/replstate.xml

Crashlytics plugin (for Android Studio and IntelliJ)

com_crashlytics_export_strings.xml crashlytics.properties crashlytics-build.properties fabric.properties

Editor-based Rest Client

.idea/httpRequests

Windows stuff:

Windows thumbnail cache files

Thumbs.db ehthumbs.db ehthumbs_vista.db

Dump file

*.stackdump

Folder config file

[Dd]esktop.ini

Recycle Bin used on file shares

$RECYCLE.BIN/

Windows Installer files

.cab .msi .msix .msm *.msp

Windows shortcuts

*.lnk

MacOS stuff:

General

.DS_Store .AppleDouble .LSOverride

Icon must end with two \r

Icon

Thumbnails

._*

Files that might appear in the root of a volume

.DocumentRevisions-V100 .fseventsd .Spotlight-V100 .TemporaryItems .Trashes .VolumeIcon.icns .com.apple.timemachine.donotpresent

Directories potentially created on remote AFP share

.AppleDB .AppleDesktop Network Trash Folder Temporary Items .apdisk