allow content to be pass as opt, and generate a new contentState

[m4gpi]

with kappa-drive, to get encryption working with our own crypto-encoder, we're having to create the content feed ourselves, as hyperdrive doesn't allow custom options when generating the content feed.

Hyperdrive used to do this, you were able to pass a _content option, but this was removed earlier in v10, so we had to pin to an earlier hyperdrive version until we could do this upgrade. Would be good to know why this was removed, (I'm guessing its to do with mounts).

Anyway, this pull-request opens the possibility of opening a hyperdrive with a pre-existing content feed, and hopefully a discussion about this. :) The PR I've got open to kappa-drive is here so you can see why we need it.

I'd like to see if we can use the _contentStates option, and made an attempt here, but I don't know if you guys are going to drop support for this private option later, so am reticent to put much time into it. Also it was difficult to figure out. I appear to have gotten it working without encryption again, but with encryption we get the following error.

      RangeError [ERR_INVALID_OPT_VALUE]: The value "-16" is invalid for option "size"
          at Function.alloc (buffer.js:367:3)
          at Object.decode (/shared/Projects/kappa-db/kappa-drive/node_modules/crypto-encoder/index.js:30:30)
          at /shared/Projects/kappa-db/kappa-drive/node_modules/hypercore/index.js:1790:17
          at Request._callback (/shared/Projects/kappa-db/kappa-drive/node_modules/hypercore/lib/storage.js:45:14)
          at Request.callback (/shared/Projects/kappa-db/kappa-drive/node_modules/random-access-storage/index.js:173:8)
          at onread (/shared/Projects/kappa-db/kappa-drive/node_modules/random-access-file/index.js:81:31)
          at FSReqCallback.wrapper [as oncomplete] (fs.js:488:5)

[andrewosh]

@kyphae Would it be easier if we extended the contentOptions function to support additional Hypercore options, like a custom encoder? If that'd solve the issue, I'd opt for that approach, since it's a smaller change.

[andrewosh]

Also, if we're going to make the ContentStates configurable, then it should be able to support configuring the content feeds of mounts too. It looks to me like this PR only lets you override the default content feed, but wouldn't apply to mounts.

Yep you're right about why that option was removed -- need to add this indirection in order to support many different metadata/content feeds when mounts are present.

Anyway, let me know if extending contentOptions would be sufficient.

[jwerle]

@andrewosh Supporting onwrite() and crypto: { ... } options would be nice :upside_down_face:

[m4gpi]

@andrewosh thanks for getting back, I'll have a proper look into it later this week and see if that'd fly.

[m4gpi]

hi @andrewosh, it'd be great to resurrect this. sorry i never got back to you, we were in a rush to meet a deadline for a first release for cobox in time for the end of our funding, so I ended up just running a fork of hyperdrive that implemented what we needed (then took a well deserved break!). we did however, end up with a bug which was related to a hypercore byte stream not closing when a feed had already closed (https://gitlab.com/coboxcoop/mono/-/issues/28) which perhaps you guys had at the time and fixed. I've resolved this for the time being in our fork by bumping hypercore-byte-stream to the latest version, but I'd like to get kappa-drive up to date with the latest hyperdrive so we're not incurring technical debt and can inherit upstream developments and fixes. That means figuring out how to setup the content and metadata feeds correctly in kappa-drive. I'm just putting this post here as a placeholder for the moment and to put this on your radar again. I'm going to review your previous suggestion of extending contentOptions function to see if this will meet the requirements and I'll put thoughts in here.

[m4gpi]

Having looked at this today, having contentOptions configurable with our own custom encoder will do the job.

The only other thing is corestore has a private function _checkIfExists which having access to as a public function would be nice so we can handle errors properly if the feed keys we are attempting to fetch are not present in our corestore.