SERVICES CONTROL (using MakeFile):
-
To Run all services :
- run all the containers that handle the services
(Crawling[golang], Logstash[elk], DB[MySQL], FE[Laravel])
using :
$ make up
- run all the containers that handle the services
(Crawling[golang], Logstash[elk], DB[MySQL], FE[Laravel])
using :
-
To Stop / Delete the containers
$ make stopOR
$ make delete -
To crawl the latest videos :
- this will crawl the latest videos subtitles and meta data from videos from youtube
$ make crawlVideos -
To index the data in your elastic search from myesql
- to be able to do proper search for users accessing your platform you need to fill your Elasticsearch documents using LogStash
$ make indexMysql - To check the results of the indexing you can run this in your [sense] or curl it from your terminal
GET http://127.0.0.1:9200/videos_en/_search
- to be able to do proper search for users accessing your platform you need to fill your Elasticsearch documents using LogStash
Useful info
-
The elastic search part is based on this docker container:
-
We wanted to use this api to fetch videos but signature is generated by youtube people, so its brought with the youtube pages as is. currently we're using alternative way using [youtube-dl]