issues
search
mwmbl
/
crawler-extension
A browser extension that can be installed by volunteers to participate in mwmbl distributed crawling.
GNU Affero General Public License v3.0
21
stars
2
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Sleep when no urls in batch
#46
daoudclarke
closed
4 weeks ago
0
Adding a new URL to a query on mwmbl.org with "Search Google" enabled redirects back to the homepage
#45
anijatsu
opened
6 months ago
0
Add Mirror to Codeberg workflow
#44
haexwise
closed
7 months ago
0
Bug fixes for curation
#43
daoudclarke
closed
9 months ago
0
Query extra search engine
#42
daoudclarke
closed
9 months ago
0
[Feature] Crawl page on demand
#41
EchedelleLR
opened
1 year ago
0
[Feature] Adjustable requests limit per website
#40
EchedelleLR
opened
1 year ago
0
Tag released versions on Git
#39
omasanori
opened
1 year ago
2
Replace the generic extension icon with official branding one on AMO
#38
omasanori
closed
11 months ago
5
Remove some logging
#37
daoudclarke
closed
1 year ago
0
Improve crawler prioritisation
#36
daoudclarke
closed
1 year ago
1
Specify the Accept-Language field to fetch requests
#35
omasanori
closed
1 year ago
1
add websites to the crawler
#34
nobaraos12
closed
1 year ago
1
Add stat ticker for pages uploaded
#33
nobaraos12
opened
1 year ago
4
add support for legacy versions of firefox
#32
nobaraos12
closed
1 year ago
1
Specify the desired language in our requests
#31
daoudclarke
closed
1 year ago
1
Internationalization
#30
adjagu
closed
2 years ago
2
Send all links
#29
daoudclarke
closed
2 years ago
0
Update README.md
#28
adjagu
closed
2 years ago
1
Build Failed
#27
adjagu
closed
2 years ago
3
Respect <meta name="robots" content="noindex">
#26
daoudclarke
opened
2 years ago
0
Add an option to pause crawling
#25
fawaf
opened
2 years ago
8
Doesn't crawl at all
#24
g00g1
closed
2 years ago
6
Detect status correctly
#23
daoudclarke
closed
2 years ago
0
Add timeout
#22
daoudclarke
closed
2 years ago
0
Bump version to 0.4
#21
daoudclarke
closed
2 years ago
0
Prevent loading big pages
#20
daoudclarke
closed
2 years ago
3
Don't try and crawl really large pages
#19
daoudclarke
closed
2 years ago
1
Crawl one page at a time
#18
daoudclarke
closed
2 years ago
1
Added popup to log crawler URLs
#17
ColinEspinas
closed
2 years ago
0
Don't send cookies
#16
daoudclarke
closed
2 years ago
0
Don't try and crawl if we're not online
#15
daoudclarke
closed
2 years ago
0
Remember visited links so we don't visit them multiple times
#14
daoudclarke
closed
2 years ago
0
Check the number of unique domains to prevent falling into loops
#13
daoudclarke
closed
2 years ago
0
Crawl more root domains
#12
daoudclarke
closed
2 years ago
0
Adapt for firefox
#11
daoudclarke
closed
2 years ago
0
Handle exceptions
#10
daoudclarke
closed
2 years ago
0
Browser Compatibility
#9
ColinEspinas
opened
2 years ago
0
Implement crawl
#8
daoudclarke
closed
2 years ago
0
Justext
#7
daoudclarke
closed
2 years ago
0
Changed dev script to build with watch mode
#6
ColinEspinas
closed
2 years ago
0
Add automated workflows to build and release
#5
ColinEspinas
opened
2 years ago
0
Respect robots.txt
#4
daoudclarke
closed
2 years ago
0
Retrieve pages
#3
daoudclarke
closed
2 years ago
0
Run a crawl iteration once every second
#2
daoudclarke
closed
2 years ago
0
Add dev mode for panel and options
#1
ColinEspinas
closed
2 years ago
1