gocolly / colly

Elegant Scraper and Crawler Framework for Golang
https://go-colly.org/
Apache License 2.0
23.2k stars 1.76k forks source link

colly.Context should be an interface? #116

Closed llonchj closed 6 years ago

llonchj commented 6 years ago

I am passing information between collectors and setting scraped data using the context. Actually i do something like ctx.Put("collector.From",X) and ctx.Put("object.Name",X). This adds some overheat to the OnScraped parsing having to classify information that could have set initially in the right structure.

I am thinking about the possibility to make colly.Context an interface so the information can be abstracted. It think it can be achieved staying backwards compatible.

What are your thoughts?

asciimoo commented 6 years ago

@llonchj you can pass an interface to the context object: https://godoc.org/github.com/gocolly/colly#Context.Put and you can retrieve it with https://godoc.org/github.com/gocolly/colly#Context.GetAny.

llonchj commented 6 years ago

I agree on the contents. Will make sense to make the Context an interface on its own?

asciimoo commented 6 years ago

I'm not sure that I understand, what would be the benefit of a context interface? And, how do you imagine the implementation?

llonchj commented 6 years ago

You're right, i will be using a value as the object.