Standardized device properties

[henrypinkard]

Based on discussions @marktsuchida and I have had and related to the camera API upgrade (https://github.com/micro-manager/mmCoreAndDevices/issues/243).

We need to come up with a better mechanism for standardized device properties. This issue to is to solicit ideas about how best to do this.

At a high level this is because:

Properties are very useful for capturing certain types of data and functionality, and it creates unnecessary work and the potential to introduce errors when you have to recreate this functionality for property-like device API calls. For example, creating methods GetX(), HasX(), GetXMaxValue() etc. for each X when all this would already exists if making X a property.

However, simply using a property for a required feature of the API doesn't work well as present. It is not clear to device adapter authors or high level code users that certain properties are required as opposed to device specific. Exposure and Binning properties are both required, but inconsistently implemented in part as a result of these shortcomings (see https://github.com/micro-manager/mmCoreAndDevices/issues/38 for more info about binning and how to fix it).

Exposure and Binning have both properties and API calls. This seems likely to have resulted from the fact that functions in MM control the generic API for devices, while properties (usually) enable functionality unique to specific devices. This duplicate functionality may have been present so that exposure could be included in config groups, to assist with switching channels etc.

The camera API has also made apparent some shortcomings of the property system. Several features related to camera triggering will be implemented following the GenICam specification as closely as possible. The most natural way to do this will be through standardized properties, and on many cameras this will create 50-70 new properties because of all the combinations of trigger types and settings. For example:

FrameStartTriggerMode: On FrameStartTriggerSource: Hardware FrameStartTriggerDelay: 5 ExposureEndTriggerMode: Off etc...

What needs to be added to properties?

A mechanism to group logically related properties. Ideally in some kind of hierarchy. For example: Triggers//FrameStart//Mode: On. A simple way of achieving this could be to designate a special character or sequence or characters to denote a new level in the hierarchy (e.g. //). This would allow UIs to group properties in a way that would be much easier to work with.

Mechanisms and conventions to discriminate:

• Required properties with a standard name (and maybe standard values) that every device of a certain type must implement. E.g. exposure for cameras. Implementing this may involve creating virtual properties in MMCore or MMDevice to throw compile-time errors for missing requirements.
• Standardized properties that a device may or may not implement, but if they do they should have a standardized name (and maybe standard values). A lot of trigger properties will fall under this category because not every camera will support them, but if they do, this standardization is needed for a future acquisition engine to automate hardware triggering
• Custom properties that lie outside of these standards and are device-specific. These should be disallowed from using names that belong to standardized/required properties.

A possible way to do this is to create functions for creating each type, which will make it clearer to device adapter developers. That is, each required/standard property will have a dedicated function like CreateExposureProperty.

On the user side, maybe also have dedicated functions for querying required properties in the mold of core.getExposure(), or something like getRequiredPropertyLowerLimit()?

[marktsuchida]

Well, I know I suggested some of these ideas, but it does feel like a lot now that I look at it. On the other hand, I'm not sure we have much of an alternative: if we use properties we need to deal with the large number of properties; if we don't use properties, not only do we lose the interop with config groups, but we will need hand-written UIs for all the trigger setting. I think we'll just have to spend some effort making this as simple as possible, unless somebody else has a better idea.

Below is a rough description of one version of what I had in mind.

It might help to first address hierarchical grouping, then required/standard properties. Once we have a hierarchical naming scheme, we can reserve names and namespaces within it for standard properties.

I suggested (off the top of my head) "//" as a possible choice of a character sequence that doesn't appear in any known device's property names (need to confirm this). There may be others that look nicer (perhaps the 3-character " > "). The goal here is to carve out a namespace (of properties whose internal name contains this sequence) that is orthogonal to all existing property names. (So we could use // only for the first occurrence and single / thereafter.)

Once we have such an indicator/delimiter (I'll keep using // and / for now), we should enforce its special meaning by rejecting property names containing // from being created via the existing API (CreateProperty() and kin). Then we can treat all properties containing // under new rules. Such new-style properties should be created with new API functions.

It is probably a good idea to use a special prefix like sys// or std// or similar to denote standard properties and groups, and entirely reserve the string before the first // for future extensions. User-defined properties in groups can be named like //Green/Intensity and //Green/Duration. (We probably need to enforce that a property must not have the same name as a group at the same level.) Standard properties can be named like sys//FrameRate or sys//Trigger/FrameStart/Delay (GUIs may choose to hide the sys// prefix or render / as something else). Standard properties should be created through dedicated functions that ideally simplify the device code as much as possible.

In the hardware config file, the hierarchical names (containing //) will be stored. Most of the internal handling of properties (including config groups and the system state cache) can stay as is. The CMMCore API should stay the same, at least in the first version, and just treat the //-containing names as regular property names. (In the future we can add functions for listing groups and group contents.) The device property browser could be enhanced to have a nicer display of property groups, but that should not be a requirement for the Core and device changes to at least work.

In general, the idea is to minimize disruption so that almost everything (including MMStudio and plugins) will just work without modification, if with a very slightly ugly display of properties. If possible, the changes should be limited to property creation on the device side and nothing else initially.

There will be existing devices that have many properties and would benefit from grouping, but renaming properties would break everybody's config. I think the best solution to this is to allow properties to have one or more "old names" registered, and to auto-replace any old names when loading a config file. (This is a feature that we can have even without this hierarchical naming scheme.) It's probably best to avoid having multiple strings referring to the same property in the running Core, so I would do the conversion during config loading only.

Importantly, all this doesn't actually solve the problem of too many trigger-related properties until the Device Property Browser (and config group editor) can actually collapse groups in the table (until then, the user will just see a flat list of all the properties). This might be the harder part, and (even though it doesn't have to happen at the same time, as I noted above) it might be good to have a plan before going forward with the MMDevice-side enhancement.

As for required properties, all that needs to be done beyond standard properties is to test that they exist after Initialize() returns.

On the user side, maybe also have dedicated functions for querying required properties in the mold of core.getExposure(), or something like getRequiredPropertyLowerLimit()?

Perhaps this can be solved by making the standard property names easier to access in the code. For example, in Java we could have a StandardProperty class that has StandardProperty.trigger().frameStart().delay() which returns std//Trigger/FrameStart/Delay. We could get rid of the parens, too. It's probably better to use something like this with the regular property APIs instead of adding a set of methods for every standard property.

[henrypinkard]

This is a great start for a plan!

I largely agree with the what of doing this, but I suggest an alternative approach on the how.

There will be existing devices that have many properties and would benefit from grouping, but renaming properties would break everybody's config. I think the best solution to this is to allow properties to have one or more "old names" registered, and to auto-replace any old names when loading a config file. (This is a feature that we can have even without this hierarchical naming scheme.)

I think auto replacing property names on-the-fly will be potential source of user confusion. This process will be difficult to make transparent to users who are making configurations but not device adapters (and many fall in this category).

Furthermore, I think the wide array of changes and improvements we are making warrant breaking of certain config files. This will be a small price to pay in exchange for making many long-needed improvements and corrections to micro-manager that will make it much better and easier to maintain going forward. For example, fixing the binning/exposure standardized properties in the correct way will necessarily break some config files. Furthermore, we've discussed moving to YAML (or some other ML) config files in the future (https://github.com/micro-manager/micro-manager/issues/1524), which in and of itself wouldn't necessarily break them, but would be a good opportunity to make forward-looking changes for how they work.

Many of the changes we've been discussing work in tandem (Triggering, signalling devices, better acquisition engine, improved memory model, direct access to devices), and are especially needed for ease of use of high performance light sheet microscopes. Thus, is will be quite difficult to cleanly break them into entirely separate projects. Inevitably, we (or at least I) will realize that, e.g., after the trigger API was already "completed", it actually needs to change to make the acquisition engine work, or something that is a standardized property shouldn't be standardized in some use case, etc. Alternatively, trying to do each of these sequentially, anticipating the needs of the subsequent ones, and periodically merging them into the main branch of the core without breaking existing functionality seems to me like a recipe for unexpected challenges that will dramatically impede progress.

I propose the following instead. We've already discussed making parallel builds of MM with the new camera API branch. I think we should expand the scope of this branch to be a place where we sequentially introduce each of these changes with the end goal of buidling out a new Major version of the core (and maybe the full software). This will undoubtedly introduce breaking changes, which we can document and allow more adventurous users to test (many of whom are already ready if we can give them a build), as well as exploring how to best build automated unit and end-to-end tests to make things robust going forward. When it is ready for general release, and having a better understanding of what has become backwards incompatible, we can write a conversion guide and scripts to help users make the switch. This will still allow us to do each feature in series (for the purposes of clarity) within this dedicated branch, but it will have the major advantage that the features do not need to moved on to the main branch and production ready for mainstream MM, and can thus have long periods of testing, improvement, and refinement before they are released for the community at large. With this workflow, we can actually see and test how all the parts fit together on a full system before finalizing how they should work.

[jondaniels]

I like the idea of making this the start of version 2.0 of MMCore which at some point will be used by the 2.0 GUI layer and not worrying about breaking existing config files. Once I needed to introduce a change to the a device adapter that broke old config files and it didn't cause much headache.

There is merit to considering how to minimize (or eliminate?) the work needed for older device adapter code be converted into the new device API. Maybe there could be a flag set or "API level" field or something so that the higher-level application knows what it can use.

I like the ideas around property grouping, having required properties, etc. If we are adding these categories consider adding a category like "advanced property" that will be hidden in the property browser by default and maybe even be omitted from metadata. That way device adapters can give an indication of which properties are likely to be used and which don't matter to most users.

[marktsuchida]

@henrypinkard

I think auto replacing property names on-the-fly will be potential source of user confusion.

What I suggested was reading old property names and converting to the current ones when loading a config file, not "on-the-fly" (whatever that means). Once a property name is updated in the device, the user will never see the old name unless they look at a config file generated by an older version (and note that only advanced users are expected to look at a raw config file). If it helps, we can also pop up a dialog and ask the user to save a new config where the property names have been updated (although I don't think this is important). I am not sure how this is confusing; it is much easier than dealing with multiple config file versions and formats -- to me that seems like the true nightmare.

Note that it is not possible to offer a config migration tool that does not involve actually loading the config, unless we store the migration data statically outside of the device adapters (which may not be a bad approach, but would be feature creep at this stage; also some properties are dynamically named and would not work with a static mapping approach).

Furthermore, I think the wide array of changes and improvements we are making warrant breaking of certain config files. This will be a small price to pay in exchange for making many long-needed improvements and corrections to micro-manager that will make it much better and easier to maintain going forward. For example, fixing the binning/exposure standardized properties in the correct way will necessarily break some config files.

I don't agree with any of these assertions. I can see a path toward all of the features we are currently discussing without making a hard break. Forking is bad, I regret having allowed MM 2.0 to be developed as a fork(s), and would not like to repeat that mistake, however temping it may be from certain viewpoints (it's not that I don't see the temptation).

I'm willing to put work into helping these changes along, including helping with breaking the changes up into manageable, incremental changes. It is very hard to collaborate if many changes are intertwined together, because then the whole picture only exists in one person's mind until it is all done (another mistake I have made in the past). We need to break things up into small units.

I would also raise the point that work needed to factor features into independent projects/units is not wasted time. It is perhaps the most important work in software engineering.

Furthermore, we've discussed moving to YAML (or some other ML) config files in the future (micro-manager/micro-manager#1524), which in and of itself wouldn't necessarily break them, but would be a good opportunity to make forward-looking changes for how they work.

I did not mean to say that changing the config file format was an immediate priority, but if we want to do it, it is much, much better not to bundle together multiple changes. It is much easier if the file format migration is independent of any change of the data schema and just be about the format. That way, we just need a mechanical converter from old to new format, and that job is done, we reap the benefits, and are ready for future changes.

Then, once we have a new format, we will have more flexibility to add new features and migration support (although it is not clear to me if any of the currently proposed projects actually need to touch the config file schema, which is why it may not be high priority).

Thus, is will be quite difficult to cleanly break them into entirely separate projects.

Which is where I think I can help. Let's not give up before even trying.

Inevitably, we (or at least I) will realize that, e.g., after the trigger API was already "completed", it actually needs to change to make the acquisition engine work, or something that is a standardized property shouldn't be standardized in some use case, etc.

We can deal with such things when they arise, and that is the only way. It is not realistic to expect that we'll get everything right every time (although I'm trying to minimize the kinds of issues that I can foresee from my experience). I do not see how making a bunch of coupled changes will prevent getting things wrong, though, except among the several features you propose to arbitrarily group together. If anything, I would expect it would increase the chances of a design problem slipping by our attention.

Also, if we need to know every detail about 2 other features before we finish the first feature, that might be an indication that those 3 features are probably too tightly coupled and need to be refined into as design where they are more orthogonal to each other. (It's a fine line -- in order to know what is "orthogonal", one does need to think about other potential/future features at some level. But the coupling, at least at the API level, should be minimized.)

If the 3 features are made to be as independent as possible, then in general there is a greater likelihood that we can later accommodate a 4th, unforeseen feature.

Alternatively, trying to do each of these sequentially, anticipating the needs of the subsequent ones, and periodically merging them into the main branch of the core without breaking existing functionality seems to me like a recipe for unexpected challenges that will dramatically impede progress.

There is no need to merge the same branch into main multiple times; that would be confusing. Ideally a "hierarchical properties" branch (PR) should be finished and merged; then perhaps a "standardized properties" PR(s). After that, the trigger API PR can hopefully be worked on with a lot fewer things to juggle in our minds.

If we have strong reason to believe that two features are inevitably interdependent, what we should do first is discuss the details and try to untangle them. There may be cases where things are truly not separable, but I suspect they are not as common as it may appear.

I propose the following instead.

This sounds an awful lot like what we were saying to ourselves when we started MM 2.0. I'm only trying to avert a repeat of that mistake (albeit at a smaller scale; but in a module where backward compatibility is much more critical).

I agreed to set up a build for the camera trigger branch specifically, because it is something that needs to be test driven with multiple concrete cameras. But if the branch is going to become a convenient catch-all for whatever change we want to make to the Core, I can only see it turning into an incompatible fork similar to MM2. Once the fork has something useful (which I think is the point), it will start to split the community.

Maybe there are measures we can take to prevent that and make it work. But it won't be easy. Clearly scoping the work seems like a minimal requirement, but doing so requires working out all the interdependencies -- basically ending up the same as breaking it up into incremental patches.

I'm not saying what you should do in your private branches. It is perfectly fine to implement several features, fully or partially, in a single branch and later extract individual features before creating pull requests. (I do this sometimes.) This could be a way to figure out how the different features will interact before committing to a design. The downside is that meaningful collaboration is much harder until the features are extracted into manageably-sized PRs.

Also, we can have one PR branch being based on another PR's branch (e.g. trigger branch based on property branch). This should only be done when absolutely necessary, but might be helpful if we want to test things out before committing to a design. By using separate branches and PRs, we can isolate the code changes for a particular feature. It will need to be understood that downstream branches may need to be rebased or reworked (possibly manually) if the upstream is changed -- which is why it is best to avoid branching off of branches, but sometimes it may make sense.

Finally, I think it will be much more rewarding to all of us if we can merge features into main one by one, as the months go by, instead of having to wait until everything is done before anything is official.

[marktsuchida]

@jondaniels

I like the ideas around property grouping, having required properties, etc. If we are adding these categories consider adding a category like "advanced property" that will be hidden in the property browser by default and maybe even be omitted from metadata. That way device adapters can give an indication of which properties are likely to be used and which don't matter to most users.

Yes, I think properties can have an enum attribute that categorizes them as Essential/Basic/Advanced/Hidden (exact categories TBD; "Hidden" (never shown in browser) can be useful for deprecated or removed properties). I think this should be done in a way that is completely orthogonal to the grouping hierarchy (there are basic and advanced trigger properties and basic and advanced motion properties, for example). It interacts slightly with the standardized/system properties, in that maybe standardized properties want to have defaults for this attribute (or maybe not) and maybe devices can override the defaults (or maybe not).

[henrypinkard]

@marktsuchida

What I suggested was reading old property names and converting to the current ones when loading a config file, not "on-the-fly" (whatever that means).

In keeping with our Code of Conduct, please try to keep these discussions polite and constructive. If something is not clear, be inquisitive and ask. Comments like “whatever that means” can be construed as dismissive and rude.

Also, if we need to know every detail about 2 other features before we finish the first feature, that might be an indication that those 3 features are probably too tightly coupled and need to be refined into as design where they are more orthogonal to each other. (It's a fine line -- in order to know what is "orthogonal", one does need to think about other potential/future features at some level. But the coupling, at least at the API level, should be minimized.)

To be clear, I’m definitely not proposing to build coupled APIs. I agree that modules should be as independent as possible. However, I feel that the easiest way to do this to build all of these in parallel rather than sequentially (a parallelized development process, not coupled features).

Finally, I think it will be much more rewarding to all of us if we can merge features into main one by one, as the months go by, instead of having to wait until everything is done before anything is official.

I agree that intermediate progress/rewards are a plus, and I think it could come just as well from a fully featured, seperate build, and an improving API.

Forking is bad, I regret having allowed MM 2.0 to be developed as a fork(s), and would not like to repeat that mistake, however temping it may be from certain viewpoints (it's not that I don't see the temptation).

I understand why you think this based on your experience with MM2.0. However, there are several important differences that make this a very difference situation than MM 2.0.

1. MM2.0 was primarily a front-end user facing component. After the fork was made there was no way to have users migrate over than by forcing them to switch. In this case, a fork can be developed independently (with rebases as needed) and once it is ready, substituted back in as a new backend

2. There will be a different development team, with different strengths/weakness and different work styles than in MM2.0. Something that is a mistake for certain programmers’ styles can be a quite good one for others. An extrapolation of the lessons of MM2.0 without considering the shifting human context would be a mistake. I'd also point out that it is extremely common for software projects to develop new versions on a separate fork so they can be thoroughly tested before making a switch, so this is by no means a paradigm that cannot work.

It is very hard to collaborate if many changes are intertwined together, because then the whole picture only exists in one person's mind until it is all done (another mistake I have made in the past). We need to break things up into small units.

There are many possible divisions of labor and ways to collaborate: multiple people writing people pieces of code, some developing and some testing, or answering questions while others do the work. Which setup are you referring to when you say this is a mistake?

I would also raise the point that work needed to factor features into independent projects/units is not wasted time. It is perhaps the most important work in software engineering.

100% agree and I’m not suggesting anything otherwise, just a different way of getting to the same endpoint

We can deal with such things when they arise, and that is the only way.

I think it is better to work them out on an experimental non-user facing branch than to expose users to a shifting API

I do not see how making a bunch of coupled changes will prevent getting things wrong, though, except among the several features you propose to arbitrarily group together. If anything, I would expect it would increase the chances of a design problem slipping by our attention.

I guess I should have said only ”I” here. I know this is how my mind works and this is the best process for me, but it is perfectly reasonable that that is not true for you or others.

There is an important precedent for the process I’m proposing: NDTiffStorage/NDViewer/AcqEngJ were all developed in parallel, all at once, and ended up at a result where they are highly decoupled, with clean, independent APIs, and are even separate projects entirely. Despite being a large project, this was all completed ~6 months. It was a simple to incorporate NDTiff into the studio recently because of the separation of features. An approach that is a “mistake” for one person’s development style can be the right choice for another’s.

A way forward

It is very important that there is a discrete point in time when we say that new device APIs are available and stable, and are ready for device adapter writers to use them. To not do this would introduce instability into the API and create more work for the writers of device adapters, because they would have to make multiple rounds of updates. I cannot see a way to make this happen, while also making this transition in a reasonable amount of time, without having a sandbox build test a fully featured version with the new APIs in place. Things have progressed at a much faster rate on the camera API when switching from planning to prototyping. If we don't want this to drag out into a multi-year project, it is essential that we are able to start prototyping the full end-to-end system as early in the process as possible.

That being said, I understand the desire to not have the community split into two different versions.

How about this as a compromise: We create an experimental build (not necessarily public, but available for select beta testers) where the full set of features/modules can be prototyped. Here, these features can be tested and we can learn more about what works and what doesn’t work. The branch will be rebased as needed to ensure it doesn’t diverge too far from the main branch.

Once we are satisfied with it, it can be split back into independent feature branches and PRed sequentially into the main branch. Maybe this requires breaking changes to configs, maybe it doesn’t—depends on what we learn during prototyping.

Let’s discuss further on Zoom.

[marktsuchida]

What I suggested was reading old property names and converting to the current ones when loading a config file, not "on-the-fly" (whatever that means).

In keeping with our Code of Conduct, please try to keep these discussions polite and constructive. If something is not clear, be inquisitive and ask. Comments like “whatever that means” can be construed as dismissive and rude.

Apologies for that. I did not mean to be dismissive, just meant to indicate that I wasn't sure what that means. But you are right, it didn't come out right.

In terms of being constructive, I find it challenging when something I propose is dismissed as "potential source of user confusion" without going into any detail about why that is the case. Don't get me wrong: I'd be delighted to be shown that my idea is not a good one, especially if we can come up with a better one from there. But that works best if some technical/logistical/strategic/whatever justification is given for throwing away an idea. I'm sure I fail to do this sometimes, too, and would be happy to be called out in such cases.

I should emphasize that developing multiple features in parallel is not what I am opposed to. I probably didn't do a good job conveying this. In fact, that is perfectly okay, and I totally agree that it is easier to design things that way. What I am opposed to is to put all those features into a monolithic "branch" or "fork". If they are indeed not coupled, why can't they just be each their own feature branch (perhaps some depending on others, but only where necessary)? If the features are kept in separate branches, then we can merge those features that we deem finished without waiting for all of the others to be done. This is really all I'm asking.

Such feature branches might arise by extracting a single feature from an exploratory (personal) branch, in some cases.

I agree that intermediate progress/rewards are a plus, and I think it could come just as well from a fully featured, seperate build, and an improving API.

Can you explain how this will work? Don't forget that we (or at least some of us) need to deal with supporting/maintaining the two variants for the whole period they exist. If we keep the main branch as the "official" one, then feature X is not useful if it is not available in the main branch. If progress/rewards are measured by what is on the fork, doesn't that mean that enough people are using the fork that we have split the community?

There are many possible divisions of labor and ways to collaborate: multiple people writing people pieces of code, some developing and some testing, or answering questions while others do the work. Which setup are you referring to when you say this is a mistake?

I'm writing from the perspective of maintaining MMCore so that we don't run into compatibility issues. I don't need to be the sole gatekeeper for this, but I think I have the most experience in that area currently. If I am to review changes to ensure that we can make a smooth transition (and that the new features are as future-proof as I can tell, within reason), it will be hard if there are 5 new features in one branch, with the commits for the 5 features intermixed. I'm worried that once such a branch exists, it will be too hard to propose any substantial changes.

Whereas if the features are on different branches (whether or not they overlap temporally), we can collaborate on getting to something we all agree upon, starting with the features that don't depend on the other ones. If in that process the feature changes, any other feature branches that depend on the present one might need to be rebased, but that is better than having to fix potentially 5 features at once to address problems in 1 feature.

In that last sense, I believe it will be less work for you, not just me, in the end, to keep features on separate branches. But if you don't feel that way, you are free to do things however you like and later present the result as separate PRs (as you propose; more on that below). It is just that in that case I worry about the fact that it will be hard for me to provide feedback until much later, when a lot more is built around that feature and therefore substantial changes require more work.

Basically I'm just asking to keep things organized so that we can work on the features one by one without having to have everything else loaded in our brains at the same time. By "work on" together, I mean hammering out the APIs and implementation, typically as a PR (but in some cases other means may be more efficient). That and I want to avoid a semi-official and log-lived branch that contains more than one topic.

I agree that there are many differences between the present case and MM2.0. In fact, MM2.0 was easier because the goal was not to merge back into the 1.4 branch. Also MMStudio had fewer things depending on it than MMCore has, and the earlier API was so poorly defined that there wasn't much to lose. (We probably would have been in a far worse mess if we had allowed MMCore to differ between 1.4 and 2.0.)

But I'm only using MM2.0 as a familiar example. If I gave the impression that this has something to do with MM or its team specifically, that is not what I intended. I think the software industry these days generally agrees that large, long-lived branches are not the best strategy if they can be avoided. Would it help if I found some articles or videos about this? I'm trying to explain in my own terms why this is, but I don't seem to be succeeding.

I do not think that the case of NDTiffStorage/NDViewer/AcqEngJ is applicable here. If you were developing a new alternative to MMCore, from scratch, it would be a whole different story. But changes in MMCore potentially affect our ~200 device adapters, MMStudio, plugins, pycro-manager and pymmcore users, and more. The MMDevice and MMCore APIs are old-fashioned and have many warts (which is part of why they are not that easy to evolve), but their stability has enabled lots of things to be built upon them and stay working for many years. This is not an excuse to leave MMCore languishing without adding new features; we just need to put effort into every change so that both the device side and the app side can update smoothly. And that is the part that I feel will be particularly intractable if not done feature-by-feature.

(There are perhaps alternative ways of thinking about how MMCore should evolve. Incremental and backward-compatible changes are the only way I can see to preserve the value that our C++ codebase offers, given how our community and codebase is structured, and that is pretty much how it has been up to now, although never formalized. I think this is a common property to any foundational component that many things depend upon. Breaking from this would be a whole different discussion at a higher level.)

I cannot see a way to make this happen, while also making this transition in a reasonable amount of time, without having a sandbox build test a fully featured version with the new APIs in place. Things have progressed at a much faster rate on the camera API when switching from planning to prototyping. If we don't want this to drag out into a multi-year project, it is essential that we are able to start prototyping the full end-to-end system as early in the process as possible.

I agree 100% with this. All I'm saying is that we need to do this for each feature (probably to different degrees), not bundle all features together into one fork. If that means we want to have unofficial automated builds for 5 different branches, that's no problem. I think I can even set up builds to happen automatically for all branches (with a list of a few to exclude) in the official repo.

Once we are satisfied with it, it can be split back into independent feature branches and PRed sequentially into the main branch. Maybe this requires breaking changes to configs, maybe it doesn’t—depends on what we learn during prototyping.

As long as that's the end plan, I'm okay with it. I just worry that it will be much more work than keeping each feature on its own branch from the beginning -- because those split-out PRs may need to undergo changes before being merged, and then the rest will need to be updated anyway. It also means it's harder for me to provide focused feedback until the features are split out. Also there is the danger of unwittingly introducing circular dependencies between the features under development (it is okay for features to depend on each other in the end, but it's more manageable if that is the result of multiple rounds of PRs that do not themselves have circular dependencies among each other). But again, as long as I'm going to have the opportunity to review and propose changes on a feature-by-feature basis, it is fully up to you how you manage your own in-progress work.

[henrypinkard]

Thanks for all the explanations. I think we are actually pretty close to consensus here.

What I suggested was reading old property names and converting to the current ones when loading a config file, not "on-the-fly" (whatever that means).

In keeping with our Code of Conduct, please try to keep these discussions polite and constructive. If something is not clear, be inquisitive and ask. Comments like “whatever that means” can be construed as dismissive and rude.

Apologies for that. I did not mean to be dismissive, just meant to indicate that I wasn't sure what that means. But you are right, it didn't come out right.

Thank you for saying that .

In terms of being constructive, I find it challenging when something I propose is dismissed as "potential source of user confusion" without going into any detail about why that is the case. Don't get me wrong: I'd be delighted to be shown that my idea is not a good one, especially if we can come up with a better one from there. But that works best if some technical/logistical/strategic/whatever justification is given for throwing away an idea. I'm sure I fail to do this sometimes, too, and would be happy to be called out in such cases.

Okay, good to know. I will try to provide more detail for this type of thing in the future.

In this specific case, what I mean is:

I think the best solution to this is to allow properties to have one or more "old names" registered, and to auto-replace any old names when loading a config file

I think this just shifts the problem to another form, which is harder to understand if you don't know what's going on behind the scenes. Existing configs will still work, but now the property shown in the browser is different than the one in the config file. I think there is a nontrivial set of users who look at config files but don't look at c++ code. I can imagine them being highly confused as to why the the property names are different between the two. However, if you have a non-existent property name in a config file, you get a pretty reasonable error on startup saying (I think) that says the property isn't there. So from a user perspective, its easy to figure out to delete it, and then you look in the property browser and find the new one and resave. You may not know why this has happened, but there's no magic happening behind the scenes to confuse you.

it is much easier than dealing with multiple config file versions and formats -- to me that seems like the true nightmare.

I don't think this will be much if at all different from when a device adapter gets updated. You often have to have a new config when this happens.

I should emphasize that developing multiple features in parallel is not what I am opposed to. I probably didn't do a good job conveying this. In fact, that is perfectly okay, and I totally agree that it is easier to design things that way. What I am opposed to is to put all those features into a monolithic "branch" or "fork". If they are indeed not coupled, why can't they just be each their own feature branch (perhaps some depending on others, but only where necessary)? If the features are kept in separate branches, then we can merge those features that we deem finished without waiting for all of the others to be done. This is really all I'm asking.

Ah I see, great! I that case I think I misunderstood your previous suggestions and There's not much (if any) disagreement here. I wasn't suggesting one branch per feature initially, but that seems quite reasonable

If the features are kept in separate branches, then we can merge those features that we deem finished without waiting for all of the others to be done. This is really all I'm asking.

Still, I do think the merges should happen fairly conservatively. Since these will be (historically speaking) pretty big changes to the low level, it will be good to be sure they're accomplishing everything we needed before declaring them finished.

I agree that intermediate progress/rewards are a plus, and I think it could come just as well from a fully featured, seperate build, and an improving API.

Can you explain how this will work? Don't forget that we (or at least some of us) need to deal with supporting/maintaining the two variants for the whole period they exist. If we keep the main branch as the "official" one, then feature X is not useful if it is not available in the main branch. If progress/rewards are measured by what is on the fork, doesn't that mean that enough people are using the fork that we have split the community?

Just that it will be rewarding to see this all come together on a separate branch, even if very are using it. Maybe not as satisfying as merging one at a time, but its still something without introducing potential problems of an unstable API.

agree 100% with this. All I'm saying is that we need to do this for each feature (probably to different degrees), not bundle all features together into one fork. If that means we want to have unofficial automated builds for 5 different branches, that's no problem. I think I can even set up builds to happen automatically for all branches (with a list of a few to exclude) in the official repo.

Awesome!

But I'm only using MM2.0 as a familiar example. If I gave the impression that this has something to do with MM or its team specifically, that is not what I intended. I think the software industry these days generally agrees that large, long-lived branches are not the best strategy if they can be avoided. Would it help if I found some articles or videos about this? I'm trying to explain in my own terms why this is, but I don't seem to be succeeding.

I think I understand what you're saying. And I definitely don't mean to suggest that the the goal here should be to have a stable alternative branch for a long period of time, but...

(There are perhaps alternative ways of thinking about how MMCore should evolve. Incremental and backward-compatible changes are the only way I can see to preserve the value that our C++ codebase offers, given how our community and codebase is structured, and that is pretty much how it has been up to now, although never formalized. I think this is a common property to any foundational component that many things depend upon. Breaking from this would be a whole different discussion at a higher level.)

I definitely don't know this for sure, but I suspect we may run into situations where there are choices to be had between doing things the correct way and keeping them as backwards compatible as possible. So I think it is a good idea to at least leave open the possibility that making such changes could be something to consider, even if that means doing a lot of updates to device adapters or leaving some device adapters/types behind. The changes we're talking about may be setting up MMCore for the next decade plus, so all options should be thoroughly considered. Of course, when given the option between backwards compatible or not, we always go with the former. But I think this is a discussion for later once more information has been ascertained.

[marktsuchida]

Just for the record, a few things I discussed with @henrypinkard on Zoom:

• For the config file, I now agree that having the file match current property names is indeed desirable. But for many (especially less technical) users breaking the config file just to rename things would be disruptive. Perhaps we can load and convert old names, but treat them as warnings and offer (if loaded by the GUI) to migrate the file. This way we get the best of both worlds.
• I think we have come to good agreement on how to proceed in terms of feature implementation (pretty much what is said above)
• Regarding the final point above (maintaining compatibility vs making breaking changes), although I lean strongly on the side of smooth migration paths, I don't think we want to prescribe a definite rule; while we all agree that it is best to maintain compatibility when possible, the strategy will have to be decided on a case-by-case basis if we run into a case where it seems worth the downside of breaking compatibility.

[henrypinkard]

@marktsuchida I'm coming back to working on this now. Do you think this a fleshed out enough for a draft PR? If not, let's meet on zoom to figure out remaining details

Features to be added + open questions

Hierarchical properties.

• Introduce the '//' characters to signify a hierarchical property, with '/' serving as a delimiter to mark different levels of the property (e.g. '//Green/Intensity').
• What to do with existing properties in existing device adapters? We agree that displayed properties should match names in config files, so seems like we can either
  • Have the GUI load and offer to convert old names in the config (e.g. 'Gain' to '//Gain')). OR
  • Allow regular properties without the '\\' to exist indefinitely as 'non-hierarchical properties' (and maybe auto convert all properties with only one level of hierarchy to be of this form?)

Standard/API properties

• Certain properties will have standardized names/be part of the API. Indicate this with a special prefix before the property name. Proposed names are 'std', 'sys', 'api'
• Standard properties should be created through dedicated functions that ideally simplify the device code as much as possible.
• Standardized properties that a device may or may not implement, but if they do they should have a standardized name (and maybe standard values). A lot of trigger properties will fall under this category because not every camera will support them, but if they do, this standardization is needed for a future acquisition engine to automate hardware triggering
• Allow for making standardized properties required, but don't yet enforce this anywhere, except in new device types/APIs that are added (i.e. new camera API)
  • Implementing this may involve creating virtual properties in MMCore or MMDevice to throw compile-time errors for missing requirements.
• Address when existing properties have the same name as standard ones. Seems like this should be limited to a compile-time (or runtime?) warning, since otherwise it would break exisiting configs. At the very least, they will be different from each other based on the presence of the prefix in front of '//'

How to create standard properties in device adapters

• A dedicated function for creating a standard property, which accepts only valid accepted keywords OR
• Each required/standard property will have a dedicated function like CreateExposureProperty.

How to access standard/hierarchical properties from Core top-level API

• Right now they're just just treated as regular properties, only with a bunch of '//' and '/' in them. How to handle them from higher/UI level can be handled in a future PR. This should prevent distruptions to existing workflows