what's the structural thought of portia's server-side?

scrapinghub / portia

Visual scraping for Scrapy

BSD 3-Clause "New" or "Revised" License

9.3k stars 1.4k forks source link

what's the structural thought of portia's server-side? #607

Closed dataknower closed 8 years ago

dataknower commented 8 years ago

Hi, can you just describe the structural thought of portia's backgroud , e.g. what's the relatiionship between slybot and slyd ?

ruairif commented 8 years ago

Slyd is a dumb webserver for handling requests, performing some validation and then persisting those changes. It has some other endpoints, one of which is used for extracting data. This is done by loading up the requested spider and requested page and then extracting data from that page using slybot. Slybot is the part that also extracts data and follows links during a crawl.