issues
search
internetarchive
/
Zeno
State-of-the-art web crawler 🔱
GNU Affero General Public License v3.0
64
stars
8
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Fix: cookie named incorrectly.
#145
NGTmeaty
closed
4 days ago
0
fix: incorrect crawl speed unpausing behavior
#144
NGTmeaty
closed
6 days ago
0
Add some live logging of the HQ consumer behavior
#143
CorentinB
closed
6 days ago
0
fix: `HQSeencheckURL()` incorrectly distinguishes between new and old URL
#142
yzqzss
closed
6 days ago
0
chore: compile regexs only once
#141
yzqzss
closed
2 weeks ago
1
Fix worker state panic when current item or URL is nil
#140
CorentinB
closed
2 weeks ago
0
Fix seencheck re-implementation with current queue
#139
CorentinB
closed
2 weeks ago
0
Add XML extractor tests
#138
CorentinB
closed
2 weeks ago
0
Add custom code for Reddit archiving
#137
CorentinB
closed
2 weeks ago
0
Allow specifying proxies per TLD
#136
CorentinB
opened
2 weeks ago
2
Allow specifying an URL for `get list` to use a remote list
#135
CorentinB
opened
2 weeks ago
0
fix: recursion present in Telegram URLs
#134
NGTmeaty
closed
2 weeks ago
0
Add job and WARC prefix to pyroscope tags for better search
#133
NGTmeaty
closed
2 weeks ago
0
Panic on /workers access
#132
CorentinB
closed
2 weeks ago
5
Verify & test our XML extraction in the context of sitemaps
#131
CorentinB
opened
3 weeks ago
0
fix: ensure there is no infinite recursion of URLs
#130
NGTmeaty
closed
3 weeks ago
0
Add additional tests and validate current URL behavior
#129
NGTmeaty
closed
3 weeks ago
0
Fix the handover bypass when seeds are loaded from list and handover is disabled
#128
equals215
closed
3 weeks ago
0
Add Pyroscope profiling support
#127
NGTmeaty
closed
3 weeks ago
0
Add proper YouTube archiving via YT-DLP
#126
CorentinB
opened
3 weeks ago
0
Correct seedList enqueuing process
#125
equals215
closed
3 weeks ago
0
Ingest seeds before starting workers
#124
CorentinB
closed
4 weeks ago
0
Send on closed channel panic
#123
CorentinB
opened
1 month ago
2
[STALE] Split Zeno in smaller packages with a better structure
#122
equals215
opened
1 month ago
3
Queue all items from seeds list before starting to crawl
#121
CorentinB
closed
4 weeks ago
0
Calming down queue stats
#120
yzqzss
closed
1 month ago
0
Change default queue behaviour
#119
equals215
closed
1 month ago
0
WAL tests fail
#118
CorentinB
closed
1 month ago
3
Enable `log` package to distribute a stored logger to all other packages
#117
equals215
closed
1 month ago
3
Lock-free AwaitWALCommitted (and smoother queue?)
#116
yzqzss
closed
1 month ago
0
Define Zeno's queuing behavior properly
#115
CorentinB
closed
1 month ago
2
Fix idna.ToASCII fail on punycode encoded URLs with port
#114
CorentinB
closed
1 month ago
0
Transform Zeno architecture to a crawling pipeline effectively making use of Go channels
#113
equals215
opened
1 month ago
0
Break down Zeno in smaller packages, especially `crawl` package which has grown too big
#112
equals215
opened
1 month ago
0
Reuse free space from popped items
#111
equals215
opened
1 month ago
3
fix: hanging on indexManager.Close()
#110
yzqzss
closed
1 month ago
2
Improve WAL concurrency performance by @yzqzss and make it optional
#109
equals215
closed
1 month ago
13
SIGSEGV logging in BatchEnqueue
#108
CorentinB
closed
1 month ago
3
Using group commit to improve WAL concurrency performance
#107
yzqzss
closed
1 month ago
1
Queue handover v2
#106
equals215
closed
1 month ago
0
Remove `runtime.Gosched()` in polling
#105
yzqzss
closed
1 month ago
1
Optimize `get list` loading performance
#104
yzqzss
closed
1 month ago
0
--exclude-host not found
#103
CorentinB
closed
1 month ago
0
Fix readItemsFromQueue() CPU 100%, fix various data races, fix hang when all items are deduplicated
#102
yzqzss
closed
1 month ago
5
Have `queue.Enqueue()` handover items to idle workers and optimize workers routines
#101
equals215
closed
1 month ago
5
Implement host rotation and Enqueue/Dequeue access regulation via atomic booleans
#99
equals215
closed
1 month ago
2
Add dequeue enqueue stats
#98
CorentinB
opened
1 month ago
2
create url_string_test.go
#97
willmhowes
opened
1 month ago
2
Restore HQ flags
#96
CorentinB
closed
1 month ago
0
Implement linkheader parsing
#95
HarshNarayanJha
closed
1 month ago
5
Next