oduwsdl / ipwb

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS
MIT License
616 stars 39 forks source link

Allow specification of a file:// URN as a source for retrieving decryption information #537

Open machawk1 opened 6 years ago

machawk1 commented 6 years ago

Related to #165 and an idea I am still hashing (no pun) out.

CDXJ output for ipwb index -e ipwb/samples/warcs/5mementos.warc then entering the key goMonarchs produces:

CDXJ output
!context ["http://tools.ietf.org/html/rfc7089"]
!meta {"created_at": "2018-08-28T18:46:10.122717", "generator": "InterPlanetary Wayback v.0.2018.08.20.0040"}
us,anothersite)/ 20161231110000 {"original_uri": "http://anothersite.us/", "status_code": "200", "encryption_key": "goMonarchs", "locator": "urn:ipfs/QmStLCMFfbyWFZ4agwspGVi53kNivcENGapxeQhYnphG6n/QmcbEyGubtekhfRxVs8VdnscweRHusFHvnCwBX59YXchcS", "encryption_method": "aes", "mime_type": "text/html", "encryption_nonce": "oC6ZMIfu1js="}
us,memento)/ 20130202100000 {"original_uri": "http://memento.us/", "status_code": "200", "encryption_key": "goMonarchs", "locator": "urn:ipfs/QmQveT4LkcXYF8Gui5AMBTbQf9mM6TjLQYjUbTSAw9rs7L/QmZPqo87xKX4nqAcRbf3VB4bvVVRc7ChNHncsCvR2wk8RN", "encryption_method": "aes", "mime_type": "text/html", "encryption_nonce": "7IvugrRv/9I="}
us,memento)/ 20140114100000 {"original_uri": "http://memento.us/", "status_code": "200", "encryption_key": "goMonarchs", "locator": "urn:ipfs/Qmcip2LjQoNyeGhByp9vN5DxxC9yHCTgfcbyhuewiUDb42/Qmc6kaRMRogE2spAimbP1RhShW2bGefHZ2MKd6BkdENeiA", "encryption_method": "aes", "mime_type": "text/html", "encryption_nonce": "Va1LMbun2ns="}
us,memento)/ 20140115101500 {"original_uri": "http://memento.us/", "status_code": "200", "encryption_key": "goMonarchs", "locator": "urn:ipfs/QmTbeL5uBhe8dNhhdkCCVvXpyY3KXRHruc8gRkpn5knNqB/QmdFq3eXyTukjni8izVftZ5hADNE3rQ4ffkfnSxqWXmuWJ", "encryption_method": "aes", "mime_type": "text/html", "encryption_nonce": "CrRjprhfKw8="}
us,memento)/ 20161231110000 {"original_uri": "http://memento.us/", "status_code": "200", "encryption_key": "goMonarchs", "locator": "urn:ipfs/QmaUzgf5ZPLC1A95kYNmhiGLYMZmMRhVJfWn1RzCixF1Kd/QmNfvFMx3f7zTzzhiLrG8tnwqUJkXAvmKMn2tCBpW7abXX", "encryption_method": "aes", "mime_type": "text/html", "encryption_nonce": "uJ1Ze161JSw="}
us,memento)/ 20161231110001 {"original_uri": "http://memento.us/", "status_code": "200", "encryption_key": "goMonarchs", "locator": "urn:ipfs/QmPX41UBUYHDVgThqDNoPQdHmCaHfS9oU851ojHpvJKvaf/QmX8sMB3mp84RkT66MvGkujdhRvxufnUb5LEXskeGjvmoC", "encryption_method": "aes", "mime_type": "text/html", "encryption_nonce": "9MM/UrdK3e4="}
us,someotheruri)/ 20161231110000 {"original_uri": "http://someotherURI.us/", "status_code": "200", "encryption_key": "goMonarchs", "locator": "urn:ipfs/QmcF6ajMonM3yZ5FRu2C8VHioFYuMDTUuSXycV2PrgqoX7/QmaT21cFbYKUeWFstLjwZzwYYqZTj97cPPzsnqPeEpLJ7N", "encryption_method": "aes", "mime_type": "text/html", "encryption_nonce": "Kt/VucsIQEU="}

Note the redundant field encryption_key (which we would anticipate being removed), encryption_method fields with identical values for multiple mementos whereas encryption_nonce varies.

A method of extracting this information may be to have it at a location in the user's filesystem like ~/Dropbox/ipwb/auth.yml/json. In this case, the expectation is that the CDXJ produced by the above ipwb indexer command may produce:

CDXJ line
...
us,memento)/ 20161231110001 {"original_uri": "http://memento.us/", "status_code": "200","locator": "urn:ipfs/QmPX41UBUYHDVgThqDNoPQdHmCaHfS9oU851ojHpvJKvaf/QmX8sMB3mp84RkT66MvGkujdhRvxufnUb5LEXskeGjvmoC", "mime_type": "text/html", "access": "file:///Users/you/Dropbox/ipwb/auth.yml"}
...

-or-

CDXJ line
...
us,memento)/ 20161231110001 {"original_uri": "http://memento.us/", "status_code": "200","locator": "urn:ipfs/QmPX41UBUYHDVgThqDNoPQdHmCaHfS9oU851ojHpvJKvaf/QmX8sMB3mp84RkT66MvGkujdhRvxufnUb5LEXskeGjvmoC", "mime_type": "text/html", "access": "urn:ipfs/someHashThatContainsDataAtAuth.yml"}
...

-or-

CDXJ line
...
!context ["http://tools.ietf.org/html/rfc7089"]
!meta {"created_at": "2018-08-28T18:46:10.122717", "generator": "InterPlanetary Wayback v.0.2018.08.20.0040"}
!meta {"access_src": "/Users/you/Dropbox/ipwb/access.yml", "id": "identiferForMetadataRecordA"}
!meta {"access_src": "urn:ipfs/someHash", "id": "identiferForMetadataRecordB"}
us,anothersite)/ 20161231110000 {"original_uri": "http://anothersite.us/", "status_code": "200", "access": "identiferForMetadataRecordA", "locator": "urn:ipfs/QmStLCMFfbyWFZ4agwspGVi53kNivcENGapxeQhYnphG6n/QmcbEyGubtekhfRxVs8VdnscweRHusFHvnCwBX59YXchcS", "mime_type": "text/html"}
us,memento)/ 20140114100000 {"original_uri": "http://memento.us/", "status_code": "200", "locator": "urn:ipfs/Qmcip2LjQoNyeGhByp9vN5DxxC9yHCTgfcbyhuewiUDb42/Qmc6kaRMRogE2spAimbP1RhShW2bGefHZ2MKd6BkdENeiA", "mime_type": "text/html", "access": "identiferForMetadataRecordB"}
...
machawk1 commented 4 years ago

@ibnesayeed We discussed this before but I cannot recall your advice for an elegant scheme to allow for per URI-R or per-URI-M authentication references in an index. Can you provide your 2-cents here. Most anything would be helpful and perhaps instigate progress.

ibnesayeed commented 4 years ago

I think shared some thoughts in https://github.com/oduwsdl/ipwb/issues/447#issuecomment-406487079 earlier.