scrapinghub / portia

Visual scraping for Scrapy
BSD 3-Clause "New" or "Revised" License
9.28k stars 1.41k forks source link

Project route does not load all spider json files #661

Closed sagelliv closed 7 years ago

sagelliv commented 7 years ago

Opening a project with many spiders in Portia became inefficient because Dulwich was opening all spider json files. Using a custom serializer SpiderListSerializer, we avoid this issue by not having including the related samples in the links property in Meta. In order to do use a custom serializer:

Additional debug logging was introduced in the Gitstorage backend to properly debug this issue.