search

infinilabs / crawler

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)
Other
305 stars 82 forks source link

does it support nofollow tag? #40

Closed kenkenchow closed 5 years ago

kenkenchow commented 5 years ago

I would like to know does it support nofollow tag?

medcl commented 5 years ago

not yet, but can be easily to add.

kenkenchow commented 5 years ago

Thanks for answering. May I know where/ how to add this feature?

medcl commented 5 years ago

check here: https://github.com/infinitbyte/gopa/blob/master/pipeline/joints/filter/parse.go#L114 just process another attribute rel, if it is nofollow skip it here or mark it . the link will be finally processed and send to further process queue, here: https://github.com/infinitbyte/gopa/blob/master/pipeline/joints/filter/parse.go#L238