include sha256 hash in asset dict

[yusefnapora]

this adds a hash_sha256 field to asset dictionaries that has the hex-encoded sha256 hash of the content. I didn't call it image_hash_sha256 because we'll eventually want to support other asset types.

Now that I'm about to open this PR though, I realized that we're resizing the thumbnail image in the writer client to 1024px max. So if you re-download from http, you may get a different hash. Maybe it's best to skip the resizing? especially since the indexer is going to resize as well. The only downside to keeping the original size is potentially having to put very large images into ipfs.

[vyzo]

I think that we want to store the original asset hash, at it's original size -- and the client to skip resizing altogether.

[vyzo]

If the indexer or any user of the client has size limitations, it should enforce them on their own in a preprocessing step. They don't belong to permanent data in the mediachain, and besides different index implementations may have different size limitations!

[autoencoder]

Long term:

Agreeing that the Client Writer -> Blockchain Core -> Client Reader parts of the pipeline should probably pass metadata & media through completely unmodified. There should be a clear stage in the pipeline at which the metadata / media data will be signed by the writing user's signature and can no longer be modified in place, right?

Factors driving us imposing media resizing / encoding restrictions somewhere in the pipeline (though not necessarily here):

• Our dataset-collection crawlers tend to almost never have access to the true originals anyway.
• The dedupe principals suggest that us achieving 100% confidence that we've obtained the true original of a media is an assumption that should never really be reached.
• We have a 100MM ingested images target in 4 weeks from now. 10MB originals would take up 1PB. So not so much just an Indexer problem - 1PB would cause unnecessary strain at many different stages of the pipeline.

BTW: Yup, the Indexer has independent steps for media resizing and re-encoding.

Have collected up some longer notes and ideas that we can hopefully all discuss in a meeting early this week.

[parkan]

@autoencoder I think there are potentially 2 relevant high-level questions:

• Do we really need (and have the legal position) to retain and serve all of the images, or even a significant % of the images, at that scale? What's the size of CC subset? What's the size of just dedupe data + small human verification thumbnail?
• (just thought of this): what if we keep the indexer normalizer flow and the blockchain ingestion translation flow separate... forever? Basically the indexer would do its own ingestion from the referenced raw sources, totally anew. This is kind of a radical baby & bathwater solution, but it does help avoid serving multiple masters (though only for the case of importing existing 3rd party JSON)

[parkan]

@yusefnapora should we pull this in?

[yusefnapora]

I actually don't know... If you run with --skip-image-downloads, then this PR won't apply at all. If you are downloading images, the client will also try to upload them to ipfs (or dynamo, if --diable-ipfs is used). Putting large original images to dynamo will hit the max size limit, and putting huge images to ipfs may not be what we want either.

That said, it would help the indexer out a lot to have the sha hashes in the records, so it can skip re-downloading. But if we're going with the "radical separation" of indexer from blockchain, where the indexer does its own normalization, etc, then I'm not sure how necessary this is.

[parkan]

Hmm yeah that's a fair point. This is a small, nondestructive changeset though, so I wonder if we can merge it just to have?

[yusefnapora]

yeah, I don't think it'll hurt anything to merge it in 👍