ibnesayeed
commented
10 months ago CDX Summary comparison of the two captures (Good and Bad):
$ zcat good/indexes/index.cdx.gz | ~/bin/cdxj2cdx.py | cdxsummary
Summarizing piped data: STDIN
CDX Overview
───────────────────────────────
Total Captures in CDX 202
Consecutive Unique URLs 202
Consecutive Unique Hosts 11
Total WARC Records Size 4.2 MB
First Memento Date Dec 01
Last Memento Date Dec 01
───────────────────────────────
MIME Type and Status Code Distribution
──────────────────────────────────────
MIME 2XX 3XX 4XX 5XX Other TOTAL
──────────────────────────────────────
HTML 3 0 0 0 0 3
Image 7 0 0 0 0 7
CSS 1 0 0 0 0 1
JavaScript 169 0 0 0 0 169
JSON 5 0 1 0 0 6
Text 0 1 0 0 0 1
Font 1 0 0 0 0 1
Other 13 0 1 0 0 14
──────────────────────────────────────
TOTAL 199 1 2 0 0 202
──────────────────────────────────────
Path and Query Segments
─────────────────────────────────
Path Q0 Q1 Q2 Q3 Q4 Other TOTAL
─────────────────────────────────
P0 0 1 0 0 0 0 1
P1 3 1 0 0 0 0 4
P2 4 0 3 0 0 2 9
P3 170 0 3 7 0 2 182
P4 5 0 0 0 0 0 5
Other 1 0 0 0 0 0 1
─────────────────────────────────
TOTAL 183 2 6 7 0 4 202
─────────────────────────────────
Year and Month Distribution
────────────────────────────────────────────────
Year 01 02 03 04 05 06 07 08 09 10 11 12 TOTAL
────────────────────────────────────────────────
2023 0 0 0 0 0 0 0 0 0 0 0 202 202
────────────────────────────────────────────────
Top 10 Out of 11 Hosts
───────────────────────────────
Host Captures
───────────────────────────────
abs.twimg.com 171
api.twitter.com 12
twitter.com 5
accounts.google.com 4
pbs.twimg.com 4
t.co 1
static.ads-twitter.com 1
dict.brave.com 1
appleid.cdn-apple.com 1
fonts.gstatic.com 1
───────────────────────────────
OTHERS (1 Hosts) 1
───────────────────────────────
Random Sample of 3 OK HTML Mementos
────────────────────────────────────────────────
* https://web.archive.org/web/20231201171243/https://accounts.google.com/gsi/button?theme=outline&size=large&shape=circle&logo_alignment=center&text=signup_with&width=300&client_id=49625052041-kgt0hghf445lmcmhijv46b715m2mpbct.apps.googleusercontent.com&iframe_id=gsi_762763_93711&as=2t8mhTvTtFQiEtW8SmlNCA&hl=en
* https://web.archive.org/web/20231201171242/https://twitter.com/?logout=1701450762854
* https://web.archive.org/web/20231201171240/https://twitter.com/ShitpostGate/status/1730213204825895325$ zcat bad/indexes/index.cdx.gz | ~/bin/cdxj2cdx.py | cdxsummary
Summarizing piped data: STDIN
CDX Overview
───────────────────────────────
Total Captures in CDX 126
Consecutive Unique URLs 126
Consecutive Unique Hosts 11
Total WARC Records Size 2.9 MB
First Memento Date Dec 01
Last Memento Date Dec 01
───────────────────────────────
MIME Type and Status Code Distribution
──────────────────────────────────────
MIME 2XX 3XX 4XX 5XX Other TOTAL
──────────────────────────────────────
HTML 2 0 0 0 0 2
Image 5 0 0 0 0 5
CSS 1 0 0 0 0 1
JavaScript 104 0 0 0 0 104
JSON 3 0 1 0 0 4
Text 0 1 0 0 0 1
Other 8 0 1 0 0 9
──────────────────────────────────────
TOTAL 123 1 2 0 0 126
──────────────────────────────────────
Path and Query Segments
─────────────────────────────────
Path Q0 Q1 Q2 Q3 Q4 Other TOTAL
─────────────────────────────────
P0 0 1 0 0 0 0 1
P1 4 1 0 0 0 0 5
P2 4 0 1 0 0 1 6
P3 104 0 2 3 0 2 111
P4 2 0 0 0 0 0 2
Other 1 0 0 0 0 0 1
─────────────────────────────────
TOTAL 115 2 3 3 0 3 126
─────────────────────────────────
Year and Month Distribution
────────────────────────────────────────────────
Year 01 02 03 04 05 06 07 08 09 10 11 12 TOTAL
────────────────────────────────────────────────
2023 0 0 0 0 0 0 0 0 0 0 0 126 126
────────────────────────────────────────────────
Top 10 Out of 11 Hosts
───────────────────────────────
Host Captures
───────────────────────────────
abs.twimg.com 103
api.twitter.com 7
twitter.com 5
accounts.google.com 3
pbs.twimg.com 2
t.co 1
static.ads-twitter.com 1
dict.brave.com 1
appleid.cdn-apple.com 1
google-analytics.com 1
───────────────────────────────
OTHERS (1 Hosts) 1
───────────────────────────────
Random Sample of 2 OK HTML Mementos
────────────────────────────────────────────────
* https://web.archive.org/web/20231201170828/https://twitter.com/?logout=1701450507965
* https://web.archive.org/web/20231201170822/https://twitter.com/ShitpostGate/status/1730213204825895325
vbanos
We try to archive a Tweet and we observe the following inconsistent behavior: We run exactly the same command on a busy server and on an idle laptop. On the laptop it succeeds but on the server the capture sometime succeeds and sometimes fails. (By failure, I mean that the capture is incomplete, the full tweet is not archived correctly, sometimes we get only the sidebar or not even that). The logs don't show any warnings or errors in any case.
Details:
We use the following command in all runs:
The laptop is a macbook air M2. The server is a 30 core, 44GB RAM machine that runs various tasks, e.g. maybe it runs 10
dockercmds similar to the one I mentioned. The memory/cpu usage could be like 60-70%. Its not overloaded.I'm attaching the crawl logs for a successful and a failed capture. bad-crawl-20231201170818604.log good-crawl-20231201171237538.log
Observations:
warningorerrorlogs in the failed capture log. Also note that there is no "timeout" or other references to errors.Hypothesis: Its like some network requests timeout (
timedRunreaches timeouts ?) and we don't capture the target URLs but there isn't any indication of that.