Docker revamp

[heyqule]

Hey Chris

Here are the list of major changes. Let me know if there are things that needed to be change.

major changes

• Dockerized the system. CLI scripts are retired.
  • All settings are fetched from src/config.yml.
• Replaced ElasticSearch 5.6 with ElasticSearch 7.3.
• Added Redis for caching.
• Automated requirements installation and kibana dashboard setup.
• Converted original scripts to modules and classes to simplify the process of building new extensions
• Data mapping have changed.
  • Each Symbol has it's own set of index. One for sentiment and one for price.
  • See src/Stocksight/EsMap for mapping details
• Sentiment and price crawlers are spawned concurrently based on your specified stock symbols.
• Improved memory footprint by spawning python instances when it's needed.

added

• Added SeekingAlpha crawler
• Added integration test cases
• Added support for generating random proxy and random user-agent.
  • may not be useful for sophisticated blockers.

current issues:

• SeekingAlpha blocks frequent accesses with 403. Follow_link is disabled for it.
• It doesn't have data upgrade migration.

[shaggy63]

Line 61 of src/StockSight/StockPriceListener.py You have D = [] which produces "list indices must be integers or slices, not str" Shouldn't line 61 be D = {}

[shaggy63]

Tweets arn't showing in Kibana. Shouldn't line 145 in src/StockSight/TweetListener.py "_id": redis_id, be "msg_id": redis_id,

[heyqule]

Partial review considering the amount of commits for the single pull request. Consider creating feature branches more frequently so Chris can approve them faster. Thank you for dockerizing.

@caseykey

Thanks for the review. However, it seems you are reviewing the older commits. There were a lot of changes and experiments on earlier commits since this is my first Python project.

This pull request was submitted after I have done a lot of changes and I asked Chris' whether he's interested on dockerizing it. I initially did this as a Python practice. Submitting the pull request wasn't planned till the later stage.