search

iipc / jwarc

Java library for reading and writing WARC files with a typed API
Apache License 2.0
46 stars 8 forks source link

CDX indexer fails to parse (webrecorder) WARC file and terminates. #78

Closed thomasegense closed 1 year ago

thomasegense commented 1 year ago

The WARC file has been made with webrecorder . cdxj(python) + warc-inder (BL) can parse the WARC-file. The error will terminate the cdx-tool indexing and write a partial last line to the cdx-file also producing invalid CDX file.

Exception from CDX inder: Exception in thread "main" java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing escape (%) pattern at java.base/java.net.URLDecoder.decode(URLDecoder.java:225)

I think this is the record that breaks it: ` WARC/1.1 WARC-Record-ID: WARC-Page-ID: 6l9q6b7tynm0ww8tezecpkc WARC-Concurrent-To: WARC-Target-URI: https://graph.instagram.com/logging_client_events WARC-Date: 2023-04-26T09:51:58.263Z WARC-Type: request Content-Type: application/http; msgtype=request WARC-Payload-Digest: sha256:7368b96670b0fdd9525fcc645fa716b687795224d678a242af020113a1f7c97d WARC-Block-Digest: sha256:db3b291b024653c5404401c321169ef68ff25c01aa6b25e20672a405796c6650 Content-Length: 2908

POST /logging_client_events HTTP/1.1 accept: / accept-encoding: gzip, deflate, br accept-language: da-DK,da;q=0.9,en-US;q=0.8,en;q=0.7 cache-control: no-cache content-length: 2228 content-type: application/x-www-form-urlencoded origin: https://www.instagram.com pragma: no-cache referer: https://www.instagram.com/ sec-ch-ua: "Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99" sec-ch-ua-mobile: ?0 sec-ch-ua-platform: "macOS" sec-fetch-dest: empty sec-fetch-mode: cors sec-fetch-site: same-site user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 x-asbd-id: 198387

access_token=936619743392459%7C3cdb3f896252a1db29679cb4554db266&message=%7B%22app_uid%22%3A%2256974635280%22%2C%22app_id%22%3A%22936619743392459%22%2C%22app_ver%22%3A%221.0.0%22%2C%22data%22%3A%5B%7B%22time%22%3A1682502712.891%2C%22name%22%3A%22instagram_web_media_impressions%22%2C%22extra%22%3A%7B%22ig_userid%22%3A56974635280%2C%22pk%22%3A56974635280%2C%22rollout_hash%22%3A%221007379527%22%2C%22frontend_env%22%3A%22C3%22%2C%22app_id%22%3A%22936619743392459%22%2C%22original_referrer%22%3Anull%2C%22original_referrer_domain%22%3A%22%22%2C%22referrer%22%3Anull%2C%22referrer_domain%22%3A%22www.instagram.com%22%2C%22url%22%3A%22%2Fregeringdk%2F%22%2C%22nav_chain%22%3A%22PolarisProfileRoot%3AprofilePage%3A1%3Avia_cold_start%2CPolarisPostModal%3ApostPage%3A6%3AmodalLink%22%2C%22media_id%22%3A%222415285276315581060%22%2C%22media_type%22%3A%22sidecar%22%2C%22owner_id%22%3A%229272148702%22%2C%22surface%22%3A%22profile%22%7D%2C%22obj_type%22%3A%22url%22%2C%22obj_id%22%3A%22%2Fp%2FCGE0Sl-BXqE%2F%22%7D%2C%7B%22time%22%3A1682502712.894%2C%22name%22%3A%22comment_impression%22%2C%22extra%22%3A%7B%22ig_userid%22%3A56974635280%2C%22pk%22%3A56974635280%2C%22rollout_hash%22%3A%221007379527%22%2C%22frontend_env%22%3A%22C3%22%2C%22app_id%22%3A%22936619743392459%22%2C%22ca_pk%22%3A%221549276785%22%2C%22c_pk%22%3A%2218121740952145782%22%2C%22container_module%22%3A%22profilePageModal%22%2C%22deviceid%22%3A%229DCBE01D-1E88-4D43-BBE1-706410D26031%22%2C%22device_model%22%3A%22Chrome+112.0.0.0%22%2C%22device_os%22%3A%22Web%22%2C%22a_pk%22%3A%229272148702%22%2C%22m_pk%22%3A%222415285276315581060%22%2C%22primary_locale%22%3A%22da_DK%22%2C%22isCovered%22%3Afalse%2C%22original_referrer%22%3Anull%2C%22original_referrer_domain%22%3A%22%22%2C%22referrer%22%3Anull%2C%22referrer_domain%22%3A%22www.instagram.com%22%2C%22url%22%3A%22%2Fregeringdk%2F%22%2C%22nav_chain%22%3A%22PolarisProfileRoot%3AprofilePage%3A1%3Avia_cold_start%2CPolarisPostModal%3ApostPage%3A6%3AmodalLink%22%7D%7D%5D%2C%22log_type%22%3A%22client_event%22%2C%22seq%22%3A7%2C%22session_id%22%3A%22187bcf98552-d8e06e%22%2C%22device_id%22%3A%229DCBE01D-1E88-4D43-BBE1-706410D26031%22%2C%22claims%22%3A%5B%22hmac.AR1ZI1sWeUop6a1CkmjJA6UcWBgb97djMyjg0Sh8zmyVpaLE%22%5D%7D `

The is the CDX-line produced by cdxj-indexer (pywb) com,instagram,graph)/logging_client_events?__wb_method=post&access_token=936619743392459|3cdb3f896252a1db29679cb4554db266&message={"app_uid":"56974635280","app_id":"936619743392459","app_ver":"1.0.0","data":[{"time":1682502712.891,"name":"instagram_web_media_impressions","extra":{"ig_userid":56974635280,"pk":56974635280,"rollout_hash":"1007379527","frontend_env":"c3","app_id":"936619743392459","original_referrer":null,"original_referrer_domain":"","referrer":null,"referrer_domain":"www.instagram.com","url":"/regeringdk/","nav_chain":"polarisprofileroot:profilepage:1:via_cold_start,polarispostmodal:postpage:6:modallink","media_id":"2415285276315581060","media_type":"sidecar","owner_id":"9272148702","surface":"profile"},"obj_type":"url","obj_id":"/p/cge0sl-bxqe/"},{"time":1682502712.894,"name":"comment_impression","extra":{"ig_userid":56974635280,"pk":56974635280,"rollout_hash":"1007379527","frontend_env":"c3","app_id":"936619743392459","ca_pk":"1549276785","c_pk":"18121740952145782","container_module":"profilepagemodal","deviceid":"9dcbe01d-1e88-4d43-bbe1-706410d26031","device_model":"chrome%20112.0.0.0","device_os":"web","a_pk":"9272148702","m_pk":"2415285276315581060","primary_locale":"da_dk","iscovered":false,"original_referrer":null,"original_referrer_domain":"","referrer":null,"referrer_domain":"www.instagram.com","url":"/regeringdk/","nav_chain":"polarisprofileroot:profilepage:1:via_cold_start,polarispostmodal:postpage:6:modallink"}}],"log_type":"client_event","seq":7,"session_id":"187bcf98552-d8e06e","device_id":"9dcbe01d-1e88-4d43-bbe1-706410d26031","claims":["hmac.ar1zi1sweuop6a1ckmjja6ucwbgb97djmyjg0sh8zmyvpale"]} 20230426095158 https://graph.instagram.com/logging_client_events application/json 200 9b7c9bb91016a0d17171d9a9307591530d2211c64f33104a1b87299a6b386f95 - - 831 6992293 webrec_regering_20230426.warc.gz

And the partial line by cdx-tool: com,instagram,graph)/logging_client_events?__wb_method=post&access_token=936619743392459|3cdb3f896252a1db29679cb4554db266&message={"app_uid":"56974635280","app_id":"936619743392459","app_ver":"1.0.0","data":[{"time":1682502720.792,"name":"instagram_web_media_impressions","extra":{"ig_userid":56974635280,"pk":56974635280,"rollout_hash":"1007379527","frontend_env":"c3","app_id":"936619743392459","original_referrer":null,"original_referrer_domain":"","referrer":null,"referrer_domain":"www.instagram.com","url":"/regeringdk/","nav_chain":"polarisprofileroot:profilepage:1:via_cold_start,polarispostmodal:postpage:7:modallink","media_id":"2408315599488034008","media_type":"video","owner_id":"9272148702","surface":"profile"},"obj_type":"url","obj_id":"/p/cfsdkcmhcty/"},{"time":1682502720.815,"name":"video_should_start","extra":{

thomasegense commented 1 year ago

And here is another webrecorder warc-file that fails. Again an partial cdx-line is written, but this time no indication of an error.

` WARC/1.1 WARC-Record-ID: WARC-Page-ID: 0jeji0c310akpwfsl1jbcsa WARC-Target-URI: https://www.instagram.com/rsrc.php/v3/y7/r/DFxAOta1VtC.png WARC-Date: 2023-03-21T18:23:44.231Z WARC-Type: response Content-Type: application/http; msgtype=response WARC-Payload-Digest: sha256:3f54ec9730ddef2706460183983edd9e3d5cc8438edb860366a588e7cd9e6d75 WARC-Block-Digest: sha256:c04a88d90799bd9abbaa141be0241712319f0a34a96c6843d7e35c2254082c90 Content-Length: 17128

HTTP/1.1 200 OK access-control-allow-origin: alt-svc: h3=":443"; ma=86400 cache-control: public,max-age=31536000,immutable content-length: 13492 content-md5: fGDFVr7WWM5O/LwSfA+8QA== content-type: image/png cross-origin-embedder-policy-report-only: require-corp;report-to="coep_report" cross-origin-opener-policy: same-origin-allow-popups cross-origin-resource-policy: cross-origin date: Tue, 21 Mar 2023 18:23:43 GMT document-policy: force-load-at-top expires: Mon, 18 Mar 2024 02:23:50 GMT last-modified: Mon, 01 Jan 2001 08:00:00 GMT origin-agent-cluster: ?0 report-to: {"max_age":86400,"endpoints":[{"url":"https:\/\/www.facebook.com\/browser_reporting\/?minimize=0"}],"group":"coep_report"}, {"max_age":259200,"endpoints":[{"url":"https:\/\/www.instagram.com\/error\/ig_web_error_reports\/?device_level=unknown"}]} strict-transport-security: max-age=15552000 timing-allow-origin: x-content-type-options: nosniff x-fb-debug: 5wf9zJTvhTlVd03KEDXnShOumOF7RGb54hB79c8VKFnztfP4m1q1ldpJEdua0ytZIHttfX7MSAkBiR320i4Q7Q== x-fb-rlafr: 0 x-fb-trip-id: 1679558926 x-wabac-preset-cookie: mid=Yj8eRgALAAGwIRC9uio6aWBZ9Q0d; ig_did=F9DD67B0-75AD-4EBC-8061-2E5333D8E680; fbm_124024574287414=base_domain=.instagram.com; csrftoken=EuyWIGONSqi30MRn2pHeVqqgXXoWBVJY; ds_user_id=3515476244; datr=TEKWYq5w-zBKbY7eNhxgFQeH; shbid="16090\0543515476244\0541710713850:01f7b344d501e0517f8e05a05b03d7e7016df3efc31280fb9f8cb13e0bc39e8bd23adf92"; shbts="1679177850\0543515476244\0541710713850:01f7fc8ecf89a372b5f3b6ec49f540f433fce514c51d9e0bdcb22dd580c54ac74ae51867"; fbsr_124024574287414=Gj7oA8CpexjkMp9QTyTDnQJREjmKc0Piw8Lbw5Jv-Mw.eyJ1c2VyX2lkIjoiMTAwMDAxNTUwNzc4MDAxIiwiY29kZSI6IkFRQ2owMDBHdXdSTmc4ck1DWVBxQ3EzekhsOG9EMGxYS25lNDFWUkZoYVU1Z1pJOUYzZmt2TGRjdXhBZGtRVy1XQXBuMGJYNjJNc2ZRaWVjLS1fR1JsOUFUY1dkVWJLZ0Y0bkltX0g5RndLbHF1RkVkM2s3Y1JoblVrUEtmbUxEWkR1aWxlTVV5bFhQZEhUNl9nd1NQZDlmUTFnbl9xbUh0ZGZ5T3NYYThjZWVGbFlJREtNb0ZQUnRtZEw4dVFwcXZ6M1k3MVNKV3pORkdybzVNNkJYSmZYQ3hCcC1HRzJDSzFmelVGWGlFblNJUk5zdDNReXFiVWdzNXFWV3ZrZnNuLUg1dC1JQVhQcnhMS21ncVdDVWlLS1ZMeUV0SnFBX2FuZnprUElTUDdxMUs1OV85cjJTTUdVRGlFSmQtLTN2WGlBZml4OWlfSm5kSEVXNlMwMm5rUUxsIiwib2F1dGhfdG9rZW4iOiJFQUFCd3pMaXhuallCQU4xenZteEQ2dDFUWkF6NFpBZ1VNWkNPQ1pCTFBBQWswR1I2WURjWkJjUUxTUWdqUFdYcm56MWt6dm5oYkRhM0p1eE1Xa3dyZ3BjR1BYY1FMWVA1a0Zra0ttRVl2YXNBVFY1QkU1ZjJRZkNJUzZRbHBjdVNjWkFGNjcxbTJaQlpBZW1ydFpCWkFjU0N1SGxVcjM0Ykh1czVLSXI4WkNGUnNXdDVBclpBaE5iSWVWaWpORU9ZS2hFeUhaQWNaRCIsImFsZ29yaXRobSI6IkhNQUMtU0hBMjU2IiwiaXNzdWVkX2F0IjoxNjc5MzI5MjY3fQ; fbsr_124024574287414=tm2pXd8O2UKGTml8kuih_yOU2q_Rn4I_29oiL1Q9Ako.eyJ1c2VyX2lkIjoiMTAwMDAxNTUwNzc4MDAxIiwiY29kZSI6IkFRQlo4aW1uTjFzaFkxX3VOTDk4TjRrOE5VUUxCdFE0V1QyX1RvSTdGNlM1clBHNXdOYVRpdVdzdFlrN3dTY1BaYVhnUUN3dTBBZEc3RmhaSk5KR3RQa1VnZGdHLVlMTFhVMUhqaUNTNFNDQ2lEYjNxQ1IwdER2eGRmbkxta1hyQ0lMWU80ck5FMTdOa3RqRWQ3bFRsRVRUbTc0MXVSTkN5NVhpOHM1Z0h2WVZBV3IybU54X0dUMWVRSWJIX2N0RTdjY3VkQlpvbXRrUDBSbWw4Z1RDeFkxMDZpdDdjTS1ILXVtRFBwaVBVbVpWWWZ1X0xxX21PTllvYnBDdXV0RVgtMjdKSmRZUXQwSWxyb3VWaW1NbnF1MGZWUVNvWnBvSlB3aVYweGlhUXFUaE1OT2V1c2JpOFQyNkZZYWJYSzN6enJhV2U3S09EUWJzaHRKTGVFN0RGSUpNIiwib2F1dGhfdG9rZW4iOiJFQUFCd3pMaXhuallCQUFDd04yVlVHQkJwU2R4YWphUmtzNTBkYXllNnZSTFpBbWc4VVk0ZmJxMmo3VVhrNUpaQmRGSzVyMUhQNFpDZjkwZE9TMnVIb2lmRHVXRnB4TExjODB1bDRzaHpNVE1kMmFhc0F4RDlaQW8wNzd6QVNoRGlUTUc2Z1hxSkVPQUM4WkNZN2tzTFVLbDJLbDJKeWhJWVBhUWgycFBnUEliWkFIeDB3ZlhqRWd1Q09DcTZodUFGb1pEIiwiYWxnb3JpdGhtIjoiSE1BQy1TSEEyNTYiLCJpc3N1ZWRfYXQiOjE2Nzk0MjI4Njl9; sessionid=3515476244%3Av2w15okEuEXWXO%3A9%3AAYc3wToa_WrPE_j6B8K26h8jwkEaRBLrCJrb24f69zZG; rur="CLN\0543515476244\0541710958876:01f734b9f4ebbd9feeb68abdfdd3d282e05540ccec8735618218f57d12dcfe481847fbef" `

And the incomplete cdx-line: com,instagram)/rsrc.php/v3/y7/r/dfxaota1

ato commented 1 year ago

Both issues are fixed by jwarc 0.28.1