search

pyk / rojak

Project Rojak, Yuk gabung diskusi di Slack nya Rojak: http://team.rojak.id/
Other
98 stars 46 forks source link

[rojak-pantau] Feat: JawaPos spider #101

Closed luthfianto closed 7 years ago

luthfianto commented 7 years ago

Work in progress, mohon jangan di-merge terlebih dahulu :smile:

Fixes: #39 Blocked by: #100

pyk commented 7 years ago

@rilut untuk jawapos yang kasus multi-page banyak gak mas? kalo dikit, kita skip dulu aja gimana mas? Kita handle yg one-page dulu (tapi make sure yg multi-page kita skip, biar gak ada invalid raw_content)

pyk commented 7 years ago

@rilut untuk handle multi-page bisa lihat teknik yang di pakai @rawgni di https://github.com/pyk/rojak/blob/c6ff60a4f8edd1b6d47081683d7f2945d0fb2a89/rojak-pantau/rojak_pantau/spiders/detikcom.py#L99-L100

yield Request(index, meta={'loader': loader, 'n': len(indices)},
                    callback=self.parse_multipage_content)

dia yield request dari loader yg sebelumnya, jadi datanya bisa di append.