create issues from my notes

[kukeiko]

What

I have lots of bits and pieces in an uncommitted document. they are growing and growing, and it gets unwieldy - so I want to have them here in this single issue, evaluate them, and then create separate issues for each item that we'll implement.

Why

So I don't forget about them and they are properly documented

List of Things

here is the raw list of things - in order from oldest to newest:

• [x] implement reductions between is-value & in-set => moved to #197

• [ ] check if current inRange(from, to, [fromInclusive, toInclusive]) way of creating in-range criteria could be more intuitive

• [x] unintuitive thingy in 01-reduction.spec.ts: => obsolete

  const from_100_to_200 = inRange(100, 200);
  const from_100_to_300 = inRange(100, 300);
  
  // [todo] it would be much more intuitive to instead have:
  // const expected = inRange(201, 300)
  // which would be valid for integer fields
  const expected = inRange(200, 300, [false, true]);
  
  const difference = from_100_to_200.reduce(from_100_to_300);
  
  expect(difference).toEqual(expected);

• [x] without thinking i've written [1-7] a lot (instead of [1, 7]) - maybe good idea to switch to that notation? it would make it easier to pick out in-range criteria in a string where there's also in-set criteria => keeping as is, as that is the usual notation

• [ ] paging on a query can only be passed to a repository if that repository can fully evaluate given criteria. e.g. if we query ProductsRepository which does not support filtering by reviews, and we issue a ProductsQuery with "where Reviews.rating > 3", we'll have to load all the Products and all the Reviews, then filter in-memory. Or we catch that case specifically and load filtered Reviews join Products, then fetch the Products and hydrate filtered Reviews, then return that.

• [ ] a property that needs to be expanded that has no link (i.e. something that lets us evaluate client-side how to join entities together) can not be put into a separate store. it has to be stored directly on the entity.

• [ ] open-api (and by proxy json-schema) allows for schema compositions that we can not cleanly map to the relational structure between entities we want to support. i feel like that is a big point to make clear to potential users.

• [ ] add vscode extensions i use to recommended list in repository

• [x] make a decision on how key index is treated. when i call "getIndexes()" - is it included or not? => moved to #212

• [ ] idea: scenario to tackle issue that with having a primary key be composited later on in the app dev lifecycle. as an example we could have a "Human" entity with SSN being the key, and then later adding "countryId" to the key. we would have "Human" and "HumanOfCountry". "Human" as key still has just "SSN", but "HumanOfCountry" has key "SSN" + "countryId". that would mean that we need a "Human" store per countryId so keys are kept unique. so suddenly a newly added, from "Human" extending schema requires us to have multiple stores. so maybe we should always consider having multiple stores per type of entity somehow?

• [ ] a relation between two entities should be able to exist without an index. an index is just a storage optimization - and, if unique is set to true - a constraint.

• [ ] i prefixed schema implementations with "Unbaked", as i plan to bake them (i.e. validation + reading all properties, keys, and relations fromm allOf() derived schemas and adding them for performant lookup). but how would I, after baking, update references to unbaked schemas (to now point to the baked schema) that exist in components like EntityStore?

• [x] api-gateway support. have an api running that receives a query, opens up a temporary workspace, delegates calls to other services, collects entities into workspace, then queries workspace. i need to consider that any service node itself might be an entity-space api, so one could not know what part of a query it can resolve without first contacting it. i currently see two options: 1st send a describe request, and then act based on it, or do both things at once: open up a stream that will at some point tell you what data you can expect to receive (that is how i had it in the now deleted old query-translation/stream code) => obsolete

• [x] expansion support (products-example) i should include expansion ASAP because im sure the merging/reduction has some bugs there. we could habe 1-n relationship from product to product-review. interesting here would also be if those reviews are to be loaded from api/products endpoint, or from a separate api/product-reviews endpoint - or both, depending on some yet-to-be-determined factor. => obsolete

• [x] paging support: top, skip, orderBy absolutely necessary. should do ASAP. => already exists as issue #21

• [ ] schema properties: they're not really done at all the only thing we're using schemas for right now is getting key, indexes and relations. we should also now think about primitive properties, and validating entities against schemas

• [ ] schema baking: bake schemas for better performance

• [ ] entity reader: class to read values based on schema, e.g. reading key/index values - also absolutely do "baking", i.e. what i had in entity-space@0.6.8 already, compiling the "copyPrimitives" based on schema data to be stupidly fast.

• [ ] adding/patching/deleting entities: user should be able to do CRUD on entities in workspace, and also persist those changes

• [ ] workspace hierarchy you have a products list UI and then click on a product to open a modal, the workspace of the modal should inherit the workspace of the list. you can now do changes in the workspace of the modal without making changes to the list already. once user closes modal, apply changes to list workspace - or discard them very easily (just destroy modal workspace)

• [ ] cache invalidation big topic where im not so sure yet on how we're gonna support that. we should be able to do interesting things though thanks to reduction mechanics.

• [ ] union support either try to shove that into the products example, or start a new example, e.g. "shapes" (square, circle, triangle) with canvases which contain them.

• [ ] schema composition via allOf/oneOf/anyOf i conveniently put that on the back-burner, but i really need to look into that. will be interesting to figure out how we'll store entities then.

• [x] blueprints: generate schemas based on blueprints, and make use of Instance type.

• [ ] open-api: import & export from and to entity-space schemas

• [ ] service-workers/worker-threads: ability to spawn service worker(s) to delegate the heavy lifting to them, e.g. filtering/expanding/normalizing entities, for that it is absolutely required that we can easily copy entities between service-worker and main-thread. also every other component needs to be easily serializable, e.g. queries, expansions, criteria, schemas, and more.

• [ ] type safety through expansions that was my driving factor back in 2018 - i absolutely wanted a reference to not be null/void 0 when i included them in a query. what i want the user to be able to do as well is say "expand this reference, but if it fails, its ok" => partially there except for "expand this reference, but if it fails, its ok"

• [ ] query toString(): now that entity-schema is part of it, what will a compact toString() representation look like? cause the schema-id is a full blown URL, and that is loooong.

• [x] reduction visualization when implementing reduction cases i had some visuals in my mind to even be able to grasp how it has to work; it would be kickass to actually be able to render them somehow (html; and if possible, terminal support <3), that would make figuring out whats going on under the hood much easier => overkill, not going to do that

• [ ] reactive queries/stores user should be able to set a query as reactive; so it'll auto update just like akita-js queries when any change to criteria/expansion/paging or even the entities themselves is done

• [x] cover more reduction/merge cases especially in the merge department i only implemented what i needed => tbd in #197

• [x] query compaction (better term to be identified) - if a query is reduced to 10 single queries, you probably just want the 1 query instead. => the way queries are dispatched to sources, this functionality just exists

• [x] render "is-value" as just the value: for that to work there needs to a possibly big change in the criteria-token-parsers, i think they would need to get greedy, but i faintly remember this being a problem w/ or/and criteria not requiring encompassing brackets. or something of the sort. also! this might remove the need of all the "binary" criteria we have, e.g. is-true, is-false, is-null, etc.

• [x] custom query options: criteria require that you have those values on the entities as properties. but that is not always possible, so we need to be able to have the user set some custom options - which might even need reduction & merging behaviour support. since those options are evaluated server-side, entities of such queries probably need to be put into different store buckets. => exists as issue #30

• [ ] support dictionaries im pretty sure that is the only common data-modelling thing that we can't support yet. needs to work nicely w/ blueprints & expansion.

• [ ] schema validate entity

• [ ] schema createDefaultEntity() (allows us to add generic to schema for nice intellisense)

• [ ] auto-generation of entities for testing purposes find some nice library that we could use to generate test data based on our schema metadata

• [ ] "search" keywords support - doing criteria with inRange() / inSet() is easy, but we really need support for queries that just have a "searchText" field, and have that be representable via criteria (if possible). probably connected w/ the "custom query options" topic

• [x] inspire yaroslav by telling him that the reduction mechanics don't just allow for client-side-caching, but can possibly open a lot of doors! one of them api-gateway! => obsolete

• [ ] normalize non-related entities, i.e. entities that have no pk?

• [x] put criteria into separate package "@entity-space/criteria"? => no

• [x] consider using prime-ng instead: while they don't use ChangeDetection.OnPush like angular/material does; it does offer a huge variety of useful controls <3 => saple apps no longer part of the repo, and for new repos I'll make a decision then (probably primeng though)

• [ ] by creating queries from payload to allow expand() to not explicitly use "queryAgainstCache()", we might create queries that can not be resolved by an API. so any component that then makes such a query relies that the data has been cached already.

• [ ] normalizeEntities() only deals with relations, but what about relations on complex types? i seem to either have forgotten about them or thought "i'll do that later" and didn't document it (or maybe i did and i just dont find it right now) => needs to be investigated if still relevant

• [ ] mergeQuery(a, b) tried to both merge & widen - to make things simpler to grasp (and have simpler code), i want to do widenQuery() as a separate call.

• 174 implement NamedCriteria.invert() so we can have "createQueriesFromEntities()" in workspace. currently the workspace specs fail

  if i add "createQueriesFromEntities()" because it is not implemented

• [ ] maybe i've written it down here already, but just to make sure: don't forget about the case where a component might issue a query for which there is no actual API endpoint. the query might make sense against workspace cache, but not against API. i am a bit puzzled here.

• [ ] document somewhere the main feature entity-space should allow for: querying data in any position in any deeply-nested component, and just getting the data. doesn't matter if from API, cache, or if in a workspace where entities are actively edited and then later committed.

• [ ] while writing example controllers, I find myself having to make POST methods just to receive expansion. would be nice if we can use GET as well

[kukeiko]

• values-criteria: I have completely ignored them so far, and there is even old, unused code in the criteria/values-criterion folder