How about add a web UI for colly?

gocolly / colly

Elegant Scraper and Crawler Framework for Golang

https://go-colly.org/

Apache License 2.0

23.2k stars 1.76k forks source link

How about add a web UI for colly? #297

Open hantmac opened 5 years ago

hantmac commented 5 years ago

Just like pyspider, how about add a web UI for Colly which can help user to control their spider.

asciimoo commented 5 years ago

This is a great idea. I started to implement a web debugger: https://github.com/gocolly/colly/blob/389bc3c811e16f7bfba497d8dc6bf4f98a51c8fe/debug/webdebugger.go , but it's really basic and I'm still not sure if it is the right approach. Would you like to work on this topic?

hantmac commented 5 years ago

First of all, we should use the separation of front and back ends;
secondly, jQuery is not a good way to build the front end, recommend Vue.js;
then,we call API which provided by Colly in front end.

How about it?

By the way, where could I find the API document of Colly?

asciimoo commented 5 years ago

First of all, we should use the separation of front and back ends;

I agree, that's why I created a debug interface which can be used by the webdebugger.

secondly, jQuery is not a good way to build the front end, recommend Vue.js;

Also agreed, it was just a quick POC.

By the way, where could I find the API document of Colly?

https://godoc.org/github.com/gocolly/colly

hantmac commented 5 years ago

Hi, I made a simple web ui long time ago (just omit the picture ), now I wanna to make it open source. And I think it is a good fit for Colly's management page.

client is the VM which one colly deploy on.
project is all your colly project. We only need to use your colly lib to make web api. What do you think? Listen to your opinions with humility.

If use wanna review the source code, I think we can put it into colly project.

vosmith commented 5 years ago

UI looks nice and clean :+1: is there a place we can play with a demo?

hantmac commented 5 years ago

UI looks nice and clean 👍 is there a place we can play with a demo?

wait a minute, I am making it and the I will push it to github.

hantmac commented 5 years ago

@vosmith The project is https://github.com/hantmac/CollyFront, I named it CollyFront, if you mind it, I can rename it.☺️😆

vosmith commented 5 years ago

@hantmac awesome I'll check it out there!

tycoons commented 5 years ago

great idea you will have a little doc or demo even brider to see the beast in action :) because I have a doubt about the implementation with go colly thank you for all his