XCrawler ( aka XGallery )

A small project ( but took so long time ) based on Laravel for web Crawling ( with porn targeted ) data

How to install

• Check installation document

Requirements

Database

• MySQL
  • Primary data
  • Telescope
• MongoDb
  • Logging

Supervisor

• Redis

Cache

• APCu

Worker / Supervisor

• Horizon
  • api : For 3rd API request
    • maxProcesses: 2
  • crawling : For Crawling purpose
    • maxProcesses : 5
  • default : General usage
    • maxProcesses : 10

UnitTest

• composer test for full test All crawler must be provided with 2 tests
• Mock
• Without mock ( request directly to target server )

Contribute

• Create PR from develop
• Make sure you have UnitTest fully
• PR must be passed all required conditions