c4software python-sitemap issues

c4software / python-sitemap

Mini website crawler to make sitemap from a website.

GNU General Public License v3.0

362 stars 110 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

--parserobots doesn't seem to work

#93 varna9000 opened 1 month ago
0
Include iframe contents

#92 marshvee opened 4 months ago
0
Client-side rendered SPAs don't work

#91 Garrett-R opened 4 months ago
0
iframe contents ignored

#90 Garrett-R opened 5 months ago
0
Add option to specify bot (user-agent) for robots.txt

#89 marshvee closed 5 months ago
1
Sort URLs alphabetically

#88 marshvee closed 5 months ago
3
Option to specify bot for robots.txt

#87 Garrett-R closed 5 months ago
2
How to exclude noindex page

#86 damarkuzz opened 9 months ago
0
Alphabetize URLs

#85 Garrett-R closed 5 months ago
1
crawling depth setting

#84 styxiik closed 1 year ago
0
How to add hreflang tags

#83 crossmania337 opened 2 years ago
2
Windows and/or Python 3.7.2?

#82 VVilku closed 2 years ago
2
Updated the string format method

#81 cyai closed 2 years ago
1
urls not saved to sitemap.xml

#80 crossmania337 closed 2 years ago
2
URL UnicodeEncodeError

#79 wkingnet opened 2 years ago
0
Exclude canonicalized pages

#78 Spidle opened 2 years ago
0
fix(crawler): add condition for not parseable content

#77 ChenKuanSun closed 2 years ago
0
Python 3.9.6 support? SyntaxError

#76 PeterWoelfel opened 3 years ago
5
AttributeError: 'NoneType' object has no attribute 'geturl'

#75 devopsenko opened 3 years ago
4
Exclude redirects from sitemap

#74 Garrett-R closed 3 years ago
2
Fixed indentation in crawler.py

#73 rstular closed 3 years ago
1
Feature Request: Limit per category/section the number of URLs to parse

#72 Veilkrand opened 3 years ago
0
Update README.md

#71 Rajeev-Kumar-DSA closed 3 years ago
3
Fixes skipping pages accessed with ?p=

#70 reuning closed 3 years ago
2
Improve Docs

#69 raghavkumarbhatia53 closed 3 years ago
1
Update LICENSE.txt

#68 singharsh0 closed 3 years ago
0
RuntimeError: Event loop is closed - with > 1 workers

#66 dpatz closed 3 years ago
6
Add support for sitemap index

#65 jswilson closed 3 years ago
4
Add package to PyPI

#64 Garrett-R opened 4 years ago
5
BUG: remove race condition in multithreading

#63 Garrett-R closed 4 years ago
0
Only limit to same domain, not same subdomain

#62 Garrett-R closed 3 years ago
0
add basic auth to enable crawling of password protected sites

#61 LoveBootCaptain closed 4 years ago
3
Handling more than 50,000 URLs

#59 jswilson opened 4 years ago
4
Stop and continue

#58 ishandutta2007 opened 4 years ago
1
No URLs found

#57 exportio opened 5 years ago
7
Fixed handling of relative URLs

#56 mnlipp closed 5 years ago
3
Fix space and tabs with pycharm software

#55 cavazquez closed 4 years ago
0
Change name project

#54 cavazquez closed 5 years ago
2
sintax error

#53 francisco-baptista closed 5 years ago
1
Added a rate limiter for load reduction on the website

#52 Bash- opened 5 years ago
7
Fix print bug in crawler report generator

#51 todpole3 closed 5 years ago
0
Limit search to path instead of domain?

#50 1kastner opened 5 years ago
5
Add multithread option

#49 Garrett-R closed 5 years ago
3
Relative URLs are parsed incorrectly

#48 ghost opened 6 years ago
2
Please move the project away from GitHub

#47 ghost closed 6 years ago
2
Error: No space left on device

#46 FranciscoPaixao closed 5 years ago
3
Video sitemap

#45 FranciscoPaixao closed 6 years ago
2
Working with Angular sites?

#44 tspicer closed 5 years ago
1
Adding trailing '/' to all URLs

#43 dgursh closed 6 years ago
1
Ignore possible errors in UTF-8 encoding

#42 ghost closed 6 years ago
1