⚗️ MetaModeling backlog

[pcrespov]

Metamodeling starts with the parametrization of a project ...

A project is parametrized by adding param-nodes and connecting them to other service nodes. These param-nodes are basically data producers implemented as constants, iterator or cycle-evaluators.

⚗️ Parametrization Nodes (or param-node in short)

• Three types:
  • constant
  • iterators
  • cycle evaluators: evaluates some output (could be from a different commit ie. space and time!) and produces a value
• As nodes, they are
  • Listed by the service catalog
  • One per type or ?
• This is a node:
  • i/o service metadata defined in backend
  • implemented in the backend
  • runs in the workflow compiler and NOT in the workers
  • is iterable -> built
• Q: can you have two param-nodes interconnected?

⚗️ Parametrized Project

• what? A project with param-nodes
• parametrized project (i.e. a project that has special nodes with parameters)
  • a parametrized project cannot run
  • needs to be iterated
  • every iteration are evaluations on parameters
  • the results of every evaluation produce a project snapshots with a given parametrization
• project snapshot : a concrete iteration of a parametrized project
  • A project snapshot is runnable
  • Has a concrete parametrization assigned
  • Has a unique name that is assigned by default but user can override (e.g. my favorite)
  • a snapshot is linked with its parent parametrized project
    • what has to change to break this link? pipeline? parametrization? names?
• how to build? A sweeper can be built adding generator param-nodes

⚗️ Project Versioning

Parametrized projects will lead to a substantial amount of "variants" of the starting "meta" project. This suggest that we need to implement a versioning system to track/identify all these variants. We follow an approach very similar to git because it is very familiar to us.

We can have:

• HEAD commit is checked out by default
• explicit commits (e.g. after some changes, we can hit "version ..." and a pop-up dialog asks you to add a message)
• can checkout a given commit
  • if modified, auto-branch
• Before a parametrized project is run, all the variants are auto-committed

[pcrespov]

• [x] remove DisplayOrder
• [x] Register py function in catalog: Implement services/catalog/src/simcore_service_catalog/services/frontend_services.py key=f"{FRONTEND_SERVICE_KEY_PREFIX}/data-iterator/number" as function
  • definition and implementation in the backend

---

• variable inputs/outputs
  • free or type contraint -> freeze to catalog
• return variable outputs as List[T] or OrderedDict[str, T]
• programmable cells

[odeimaiz]

@KZzizzle says:

Just a couple of questions:

1. Are you committed to the term "snapshots"? The word to me is more related to "screenshot" than a version of the study. Something like "save version" and "All versions" and "save version as: version name" may be more intuitive and more in line with git vocabulary which can ease transitioning between the two.

2. If you have "unsnapshotted" changes and then you click on an older "snapshot" is there any kind of warning that your changes will be deleted? (I assume they will be deleted and not autosaved somehow)

[odeimaiz]

@ignapas says:

Comments from a dumb user:

• What are snapshots? What is their purpose? I am also with Katie in favor of calling them versions... and the user could browse the "version history"

• I found it a bit annoying that it is mandatory to give them a name, and that it defaults to the study name (maybe make it default to an ISO timestamp new Date().toISOString() // "2021-08-30T15:30:23.536Z" and assign it a name later on?) edit: I like the default name when no name is provided, but it has the problem of not updating if the user changes the name of the study (last element between [])

• When I open a snapshot then this one is available in the Dashboard with name <orig_name><snap_name>

  • it will be usually two times the same name (the ISO timestamp would make this look better)
  • from this "snapshot study" I cannot access the other snapshots nor the original study. maybe, instead of "opening" the snapshot, the user could "apply" it and then the study in the dashboard would always load to that version, without duplicating studies?

• should the user be able to delete and rename snapshots?

• Some UX feedback

  • A snapshot is created successfully
  • A snapshot preview is loading

• When WEBSERVER_DEV_FEATURES_ENABLED is false the Take Snapshot button is visible but the operation fails

• This weird behavior

• MINOR: Button disabled for empty studies?

[KZzizzle]

The top bar is getting very crowded (see also Clusters in frontend #2517 (comment)). It may be close to a point where there needs to be some consolidation of study operations:

• guided mode
• run options: selected node, clusters
• export/import
• versioning

[KZzizzle]

For the snapshots frontend:

• consider color-coding the "git graph" to differentiate between iterator snapshots vs other snapshot varieties
• currently, iterator snapshots show up as "branches" of increasing length in the git graph. This isn't necessarily intuitive since they should be happening in parallel. Increasing length may make sense for optimizer iterations