Modular Crawlers Based - Githubissues

ApolloCrawler / microcrawler-webapp

Microcrawler Web Application written in Elixir / Phoenix

MIT License

2 stars 3 forks source link

Modular Crawlers Based #52

Open korczis opened 8 years ago

korczis commented 8 years ago

Currently Microcrawler expects / supports only HTTP(S) Crawlers.

It will be nice if the Architecture will be modular (or extension friendly) and allow extension of supported protocols / schemas.

[x] HTTP(S)
- [x] http(s)://hostname
- [ ] http(s)://username:password@hostname
[ ] SQL - MySQL, PostgreSQL
- [ ] mysql://username:password@hostname/database?query="..."
- [ ] postgreql://username:password@hostname/database?query="..."
[ ] NoSQL - Mongo
- [ ] mongodb://username:password@hostname/database?query="..."
[ ] Twitter
- [ ] twitter://query="..."