-
Hi,
@nramirezuy and me were debugging memory issue with one of the spiders some time ago, and it seems to be caused by ImagesPipeline + [S3FilesStore](https://github.com/scrapy/scrapy/blob/master/sc…
kmike updated
7 months ago
-
Why is the crawlable feature removed in 1.6? Any further explanation would be awesome. ;)
-
### Steps to reproduce the problem
1. Share https://www.jefftk.com/test/no-robots or another link which prohibits crawlers via robots.txt
### Expected behaviour
Mastodon instances fetch https://w…
-
undetected chromedriver worked well till yesterday but now, cloudflare improved and the chromedriver is not bypassing cloudflare. I have attached the screenshot of it. cloudflare is just looping the c…
-
Hi.
Is it possible to create a sitemap for the docsify site?
-
We currently support a subset of MediaWiki-style links. I am implementing support for the MediaWiki links with a "friendly label", eg:
```
[[My Page|Click Here]]
```
would become:
```
Click Here
`…
-
**Context** - This is a proposal from Google based on our experience consuming schema.org markup and working with similar data from online merchants. If it were accepted, it would make it easier for u…
-
Testing this product, so far so good. I will have a number of sites that use OKTA SSO which I will need to crawl. Any pointers on how to do this?
-
## Feature request
#### What problem does this feature solve?
It improves SEO of pages generated by Docsify. It makes it easier for search engines to find the relevant content, and when a user s…
-
**Dates**: 16-27 January 2017
**Sprint Milestone**: https://github.com/ipfs/archives/milestone/1
**Waffle Board**: https://waffle.io/ipfs/archives
**Participants from IPFS Team**:
* @flyingzumw…