-
https://www.idealista.it has a `robots.txt` which appears complex but essentially has the following: -
- `User-agent: *`
- `Allow: /en/geo/`
**Scrapy** (1.6.0) keeps telling me that where-ever …
-
Add a 'deploy to Heroku' button for one-click hosting of an instance on the Heroku platform.
-
I've got this running on heroku. But in [Acceptable Use Policy](https://www.heroku.com/policy/aup) line 21 of ```Prohibited Actions``` , they say we are not allowed to use such tools on heroku for scr…
-
Hi,
According to your example at https://github.com/EricChiang/pup#--and- pup is supposed to print :
```css
$ cat robots.html | pup 'title, h1 span[dir="auto"]'
Robots exclusion standard - Wik…
sebma updated
5 years ago
-
(continiung the discussion in https://github.com/pandas-docs/pandas-docs-travis/pull/3#issuecomment-503148283)
Afaik we're currently not telling search engine which is the copy of the documentation…
-
Hi David,
First, thank you for creating all those useful plugins for Kirby CMS :)
I want to insert specific content to my robots.txt file. As described in your README, I added the following entr…
Uelii updated
5 years ago
-
### Steps to reproduce the issue
install A12 on php 7.3.16 (newest is 18)
login into admin > than this pops out
Error
RuntimeException: Could not connect to statistics server: Operation timed out…
-
**Original bug ID:** 6766
**Reporter:** braibant
**Status:** closed (set by @damiendoligez on 2015-02-24T21:13:03Z)
**Resolution:** fixed
**Priority:** normal
**Severity:** trivial
**Category:** web s…
-
The current approach for filtering collisions based on collision groups is to include the groups in the urdf and parse them with matlab. I can't find an equivalent procedure to filter collision when u…
-
Hi,
sorry, if I ask, but I don't understand why the robots.txt contains "GetSitemap: {sitemapUrl}" instead of simply "Sitemap :{sitemapUrl}?
it's about this function in the RobotsTxtFactory:
```
…