dataset-controller: Recover from failed/partial uploads

Several thoughts:

I don't want to do alpha/beta versions of CRDs if we don't have a process for alpha/beta development. I prefer monotonic versioning in the absence of process.
Which of the eight falsehoods are we invoking, exactly? From the list, the only one that looks relevant is "the network is secure," and I'm only assuming that the k8s fabric is secure.
I wasn't going to make hundreds of custom objects without customers. I imagined that an individual customer may have several dozen DataSet objects, along with SimilarityIndex objects, LanguageModel objects, and eventually RAGPipeline objects. It's starting to sound like this data should be in a durable database which supports relational queries (SQL) instead of k8s.
I don't understand what DataSetBroker would do for us, exactly. As a side table for SQL, it makes plenty of sense; as a CRD, I don't know what I would do with most of those fields.
On Discord, we agreed to use JSON to store k8s objects in git. I'm okay with using YAML, but YAML is literally a superset of JSON syntax, and I'm not sure which parts of YAML are essential to the developer experience here.
We don't have an internal storage space which would fit URIs well. (Aside from using k8s as a database, of course.) I'm not opposed to URIs for designating objects, particularly in a content-addressible scenario, but I don't think that we have that sort of infrastructure yet.

artificialwisdomai / origin