Closed hannakoppelaar closed 1 year ago
Forgot to add documentation for the cdxIndexer.format option, but now this PR is really ready for review :)
Hi @hannakoppelaar, The parameter "String urlkey" of a line of the CDX is not passed from CDXIndexer explicitly. A piece of CDX file looks like this:
com,google-analytics)/analytics.js 20221106221337 https://www.google-analytics.com/analytics.js text/javascript 200 VZD42NGL3YEBUSGX7EX4QCVPA2QTQEMT - - 51199 249944 IAH-20221106221320901-00000-12216~dev~8443.warc
com,imrworldwide,secure-nz)/v60.js 20221106221338 https://secure-nz.imrworldwide.com/v60.js text/html 301 DFS4JFJMZDAIFJRQP3LHAYNFPKVWMMX2 https://cdn-gl.imrworldwide.com:443/v60.js - 700 303332 IAH-20221106221320901-00000-12216~dev~8443.warc
Would you mind take a look is it the expected contents?
Was this file
webcurator-store/src/test/java/org/webcurator/core/store/warc/14055/1/_resource/00000000.jdb
intended to be part of the commits?
No, it was being generated by a test, I've removed it and added it to .gitignore
Hi @hannakoppelaar, The parameter "String urlkey" of a line of the CDX is not passed from CDXIndexer explicitly. A piece of CDX file looks like this:
com,google-analytics)/analytics.js 20221106221337 https://www.google-analytics.com/analytics.js text/javascript 200 VZD42NGL3YEBUSGX7EX4QCVPA2QTQEMT - - 51199 249944 IAH-20221106221320901-00000-12216~dev~8443.warc com,imrworldwide,secure-nz)/v60.js 20221106221338 https://secure-nz.imrworldwide.com/v60.js text/html 301 DFS4JFJMZDAIFJRQP3LHAYNFPKVWMMX2 https://cdn-gl.imrworldwide.com:443/v60.js - 700 303332 IAH-20221106221320901-00000-12216~dev~8443.warc
Would you mind take a look is it the expected contents?
Hi @leefrank9527, the default CDX format that we generate is N b a m s k r M S V g, which contains this sequence of fields:
N: massaged url b: date a: original url m: mime type of original document s: response code k: new style checksum r: redirect M: meta tags S: compressed record size V: compressed arc file offset g: file name
(See https://iipc.github.io/warc-specifications/specifications/cdx-format/cdx-2015/)
It seems to me that those two lines conform to this format or am I missing something? Note that the redirect field is missing in the first line, but that's not unusual.
something Hi @hannakoppelaar Thank you for the response. It looks good to me now.
Fixes issue #70 and allows the user to specify the desired cdx output format.