TeamHG-Memex undercrawler issues

TeamHG-Memex / undercrawler

A generic crawler

78 stars 25 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

How to store urls and html content to json format?

#83 AlexPapas opened 3 years ago
1
How to set splash.plugins_enabled for Undercrawler.

#82 nehakansal closed 5 years ago
3
Blank pages extracted in a crawl.

#81 nehakansal opened 6 years ago
5
Where are debugLogs logged when splash.args debug is true?

#80 nehakansal closed 6 years ago
2
Lua error.

#79 nehakansal closed 6 years ago
1
How can i get both cookie and html through def parse(self,response)

#78 bswbatman closed 6 years ago
2
Question about Downloader Middlewares

#77 nehakansal opened 6 years ago
4
Memory problems: SplashRequest references keep going up

#76 nehakansal opened 6 years ago
9
Undercrawler concurrency and Splash slots

#75 nehakansal opened 6 years ago
2
Config/issues with running multiple crawls?

#74 nehakansal closed 6 years ago
2
What is the location of CDRv2 exports?

#73 arjunv opened 7 years ago
1
crazy form submitter is not using form url

#72 kmike opened 7 years ago
0
updated from orig

#71 thebennos closed 7 years ago
1
Accept multiple URLs from the command line

#70 lopuhin closed 7 years ago
1
Use pathlib instead of codecs

#69 lopuhin closed 7 years ago
1
Add FOLLOW_LINKS option

#68 lopuhin closed 7 years ago
1
CDR v3

#67 lopuhin closed 7 years ago
1
Don't canonicalize file URLs: scrapy 1.4 compatability

#66 lopuhin closed 7 years ago
1
test_documents fails on scrapy master

#65 lopuhin closed 7 years ago
1
Update scrapy

#64 lopuhin closed 7 years ago
2
More screenshot options, save screenshot path to item metadata

#63 lopuhin closed 7 years ago
1
EvalError: Refused to evaluate a string as JavaScript

#62 lopuhin opened 7 years ago
5
feature request Soft404

#61 thebennos opened 8 years ago
0
Creating a working docker image

#60 thebennos closed 7 years ago
5
S3 Filestorage

#59 thebennos closed 7 years ago
4
Bad interaction of subdomains and autologin keychain

#58 lopuhin opened 8 years ago
0
Redirect from domain to www.domain is not handled correctly without splash

#57 lopuhin closed 8 years ago
1
Dockerfile for running undercrawler with arachnado

#56 lopuhin closed 7 years ago
1
Simplify autologin installation on travis

#55 lopuhin closed 8 years ago
0
Do not fail too early - return error pages as well

#54 lopuhin closed 7 years ago
0
Use new settings variable names from autologin-middleware

#53 lopuhin closed 8 years ago
1
Optional splash support

#52 lopuhin closed 8 years ago
1
Lua page script timeouts when trying to render binary pages

#51 lopuhin opened 8 years ago
5
An option to run without splash

#50 lopuhin closed 7 years ago
2
Use dupe predictor and utils from MaybeDont

#49 lopuhin closed 8 years ago
0
External autologin middleware

#48 lopuhin closed 8 years ago
0
Continue exploring possible duplicates

#47 lopuhin closed 8 years ago
1
Non-blocking autologin

#46 lopuhin closed 8 years ago
3
Long delay for the first request

#45 lopuhin closed 8 years ago
1
Do not create items for document urls we already fetched

#44 lopuhin closed 8 years ago
1
Fix domain regexp

#43 lopuhin closed 8 years ago
2
DupePredictor should assign more weight for recent samples

#42 kmike closed 8 years ago
1
don't always ignore duplicate pages

#41 kmike closed 8 years ago
1
download out-of-domain iframes

#40 kmike opened 8 years ago
3
increase aggresiveness for file downloads

#39 kmike opened 8 years ago
1
fragment is removed from pagination links

#38 kmike opened 8 years ago
0
issues with allowed domain regexp

#37 kmike closed 8 years ago
0
confusing `WARNING: Dropped` lines in log

#36 kmike closed 8 years ago
1
spider can't be stopped with Ctrl-C when autologin is pending

#35 kmike closed 8 years ago
3
Cache lua_source and js_source

#34 kmike closed 8 years ago
1