yasserg crawler4j issues

yasserg / crawler4j

Open Source Web Crawler for Java

Apache License 2.0

4.54k stars 1.93k forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Question Regarding the Crawler Logic

#475 vadimlevitzky opened 1 year ago
3
Authentication OpenID Connect not working with Crawler4j

#474 faiezkharrat opened 1 year ago
0
when will release crawl4j new version?

#473 epubreader opened 1 year ago
1
[Performance Bug Report] Specific input cause endless loop in UrlResolver.java

#472 yinxiL opened 2 years ago
1
Can't fetch content of random pages

#471 NingMorris opened 2 years ago
1
Could not find artifact Sleepycat

#470 nbngoc93 opened 2 years ago
2
Cannot fetch content of some website but python can.

#469 ryan701212 opened 2 years ago
1
Fix Maven dependencies in build.gradle

#468 ytxmobile98 opened 2 years ago
0
Not working with JDK 17

#467 mehikmat closed 2 years ago
9
examples or code scraps of how to use crawler4j when pages contain js generated links? ...

#466 Albretch opened 2 years ago
0
Improve GRADLE build Performance

#465 i-Taozi closed 1 year ago
0
Upgrade to docker-compose-rule-junit4 1.7.0

#464 uarlouski opened 3 years ago
1
Improve Travis CI build Performance

#463 chenzhang22 closed 2 years ago
0
what is the purpose of threadShutdownDelaySeconds and cleanupDelaySeconds

#462 plee0606 opened 3 years ago
0
Synch

#461 ZeeBeeGit opened 3 years ago
0
Units for maxDownloadSize are not defined in docs/comments

#460 Original-Carl opened 3 years ago
0
Upgrade to Gradle 6.7.1

#459 valfirst closed 2 months ago
1
Handle escaped * and $ characters in RobotsServer

#458 Mideen opened 3 years ago
0
Enable Dependabot on the repo to keep dependencies up to date

#457 sellersj opened 3 years ago
1
Q: any plan on releasing a new version with updated dependencies..

#456 casperrafn opened 3 years ago
4
Wrong comment

#455 samuellembke opened 3 years ago
0
Fixed typo

#454 teeteejo closed 3 years ago
0
add jetty server response some info

#453 linweisen opened 4 years ago
0
config.setResumableCrawling(true) make it too slow

#452 linweisen opened 4 years ago
0
Typo

#451 LondonWebFactory closed 3 years ago
0
Even if seeded with different domains, crawler4j crawls one domain at a time

#450 theomails opened 4 years ago
0
Fix processing of backslashes in a path

#449 ikalinin1 closed 2 years ago
2
Backslashes in path processed incorrect

#448 ikalinin1 opened 4 years ago
0
When I set depth to one level, I can see that only one thread is working

#447 basilmusa opened 4 years ago
0
added disregarded protocols

#446 mihalispap opened 4 years ago
0
Changed BasicCrawler to WebCrawler

#445 SpyrosKou opened 4 years ago
0
Selenium basic integration

#444 dgoiko opened 4 years ago
1
Future - Sustainability of project

#443 papadako opened 4 years ago
2
Performance Issues

#442 papadako opened 4 years ago
5
EMBED tag is not parsed causes we missed embedded URL

#441 Mideen opened 4 years ago
0
URLcanonicalizer returns invalid URL as a valid one

#440 Mideen opened 4 years ago
0
Crawling https://developress.netsons.org returns 0 pages

#439 loxal closed 4 years ago
4
configurable slf4j logger

#438 leder11011 closed 4 years ago
3
Retry on ContentFetchError

#437 dgoiko opened 4 years ago
0
URL's with spaces are not getting considered.

#436 RohitGoyal08071996 opened 4 years ago
1
Strange conditions to follow redirection

#435 dgoiko opened 4 years ago
0
Generic crawl controller

#434 dgoiko opened 4 years ago
2
Configurable inmediate redirection

#433 dgoiko opened 4 years ago
0
Base clases provide more protected methods for subclasses

#432 dgoiko opened 4 years ago
0
Cloneable CrawlConfig

#431 dgoiko opened 4 years ago
0
No DocIDs will be created if maxPagesToFetch is reached (most times).

#430 dgoiko opened 4 years ago
0
Timeoutable regular expressions in RobotstxtServer

#429 dgoiko opened 4 years ago
0
Granularity in exception

#428 dgoiko opened 4 years ago
0
use useSystemProperties() for HttpClient builder

#427 akacase opened 4 years ago
0
Configurable database names

#426 dgoiko opened 4 years ago
2