Request for Requirements: ISIS Release Process and Long Term Support

[jlaura]

Overview:

For the few months, we are actively soliciting input from the community of ISIS users to help determine the pace and requirements of ISIS releases. This request will be used, by members of the ISIS TC, to develop a plan for the ISIS release cadence and to determine if and how a long-term support (LTS) model may be implemented by the ISIS project.

Request:

We are soliciting user feedback on the ISIS release cycle in order to better define release requirements. Specifically we want to know how you interact with ISIS releases, the life-cycle or update cadence that you use with ISIS, and your expectations for release stability through one or more user stories. In the context of this request, user stories are at most five sentences summarizing generally who you are (a persona), what release requirements would fulfill your needs, and why this story fulfills your needs.

How can I participate?

• Write a user story.
• Add an emoji to an existing user story to let us know that you agree or disagree with it.

On 'stability'

I added this section because we are seeing the word stability or proxy phrases for stability (e.g., should always work) being used. When writing about 'stability', please consider that that word has different meanings for different users. One resource to help guide contributors in writing more specific user stories might be the ISO:25010 specification on software stability. Lots of other options exists too! The primary goal here is to have text that helps the ISIS TC write specific requirements to support a large subsection of the user base.

User Story Templates:

At the most simplistic, you could use the following template as a mad-libs style starting point:

As a \<persona>, I \<want to>, \<so that>.

Some more fuller examples:

As a cartographer, I work on inner body planets creating large scale grounded mosaics. I keep a consistent version of ISIS during a project (1-12months in duration), since changes to the software can cause me to have to reprocess data. If bugs are present in the version I am using for the project that impacts my work, I need to have a way to get bug fixes. I am interested in updating my ISIS version in between products.

As an individual scientist I am looking at the chemical make up of various interesting sites on Mars. I am publishing scientific theories I need to output of ISIS to be consistently accurate or have an explanation of any improved metrics, which might help explain differing values between ISIS versions. My projects range from 3-6 months in length and culminate with a publication(s). I sometimes follow up or expand on my work a year or more after the initial publication(s).

Please consider if you have more than one user stories within your work and feel free to break them up. Additionally, it is okay to have different requirements/desires for different tasks. We are hoping to satisfy as many needs as possible so don’t be afraid to share conflicting desires.

Next steps:

A subset of the ISIS TC will review and synthesize all presented user stories with the goal of identifying a release cadence and long-term support (LTS) model that supports as many users as possible.

[rbeyer]

As an individual scientist, I want to use ISIS to perform data processing for a particular project that will lead to a paper. During the lifetime of this work (six months to a year), I want the output of ISIS to be dependable and reproducible, but if there are bug fixes, I want those. In the past, new versions of ISIS would sometimes introduce non-reproducible behavior that wasn't necessarily just a bug fix, and so I have become wary of upgrading ISIS in the middle of working on a project. However, if I could be guaranteed that there were version upgrades during the lifetime of my project that were truly just bugfixes, then I would be more likely to upgrade during that lifetime. Otherwise, I would only upgrade ISIS before starting a new project.

[rbeyer]

As a developer of a software package that depends upon ISIS (e.g. Ames Stereo Pipeline), I need to develop against a version of ISIS whose API and underlying data structure is unlikely to change significantly over the average time span between releases of my software. My project is a much smaller enterprise than ISIS, and we are only able to release about once a year. If the ISIS API and data structure changes significantly on a shorter time scale (this probably means major version changes of ISIS and its data), my users complain about functionality not working, but my team can't do anything about it until we can perform another release.

Ideally, we could get our project release schedule synchronized with major version changes to ISIS (if we knew what that schedule was and it wasn't faster than our release cadence). We'd develop against an ISIS release candidate, wait for that RC to be made an official release, do a final check, and then make our own release, so that the lifetime of our release matches (as much as possible) the lifetime of the ISIS revision we are baselined against.

Having ISIS software and data versioned and available for a time scale approximately similar to the release cadence of my software project would alleviate this problem (even if the API and data structure resulted in major rev ticks on a timescale faster than my project's release cycle), as then we can direct our users to a specific version of ISIS and its data structure, and they should be able to get by until our next release.

[rbeyer]

As a spacecraft mission IT manager, I need different things from the ISIS release pattern. (1) During the development stage of my mission, either ASC developers or my own developers will be creating new features and finding new bugs that need to be addressed and merged as quickly as possible so that ISIS is in a good state for "my mission." (2) After this phase, I now need ISIS to be rock-solid and provide reproducible results while my mission operates. I understand that bugs will be found and features will be added. Ideally, I'd like to get the bug fixes, but would probably prefer to live without the features (because they sometimes have unintended consequences and I want to keep the changes minimal because those changes might ripple out into my other data processing systems and require a lot of patching).

I appreciate that my mission may be active for years or even decades, and it is unreasonable to expect only bugfixes for that time, and the mission is likely to freeze on a particular major version of ISIS. The longer that version can receive bugfixes, the better. Likewise, knowing the approximate time that a major version would be getting bugfixes will help my staff and I plan for changes. We'd much rather be able to see a major revision coming well in advance, so that we can plan for workforce to accommodate the change, and test against the RC before it becomes a release. Otherwise, we might stay frozen on a particular version for many years.

[astrostu]

As an average scientist user, I would want to have access to the latest ISIS version to use to do whatever I need for my particular project, be it processing one LROC-NAC image, or ten Cassini ISS images. I would hope that new versions are more stable, are more correct (in case inaccuracies in calculations were found that have been fixed), and have more/new features or capabilities I might want. My projects typically require ISIS only for the initial data processing, and afterwards it does not matter if something is forwards or backwards compatible because I don't tend to reprocess my data in the middle of a project.

[astrostu]

As a developer who writes code that interacts with ISIS and expects that ISIS behave in a certain way, including various data structures and output files, I need ISIS to remain stable and I need changes to be documented. For example, if a cube header format changes, and my code expects a certain format for a certain line, that could break my code. As another example, if the way that ISIS writes control networks moves from its current version (5) to a new one, my code would be completely broken, and an absence of documentation would make debugging it very difficult; there is no documentation about what a version 5 control net is, versus the past version (2). So moving forward, extremely detailed documentation about how things have changed from one ISIS version to the next would greatly assist me in understanding what I need to change in my own code to make sure it still works.

[jessemapel]

@astrostu Your two comments are your experiences with releases in the past. These can be helpful when writing user stories, but are not user stories themselves. Think about how you would like to interact with releases.

For example one story I could see being informed by your first comment is: "As a scientist I want to have access to the latest software version with all available bug fixes and features so that I can use it to process a small dataset once and then use it in other software."

For the times when installing or updating a new version resulted in problems, think about how you expected it to go. That expectation is the "what" part of the user story.

[astrostu]

@jessemapel I've edited.

[jlaura]

I added a section to the original post about 'stability'. The user stories already written could possibly benefit from more specificity in the 'stability' department. Please see the link.

Also, I am going to hide the comments in here that are not user stories (this one included in a day or two) to keep the discussion really focused.

[lavalaz]

Scenario #1: A scientist who is focused on getting a task done and is looking for a fast and furious rate of small releases so their work does not stall.

Story: I am a researcher processing multiple data sets for a section of planet X. I build a pipeline using seven isis applications mixed with some unix utilities and process 289 images from one of the instruments. I discover an obscure issue with isis (maybe a bad interaction between special pixels and a calibration routine). I report it, work with the developers over the next few days, get a fix, and keep processing the following week. Brilliant publication follows shortly thereafter with glowing acknowledgement to isis and its amazing support team.

Note: I am aware of very real and practical problems with providing this kind of support that is driven by the user's schedule with no consideration of other work the developers are doing. But I think it is a real use case that is worth keeping in mind.

[michaelaye]

sorry, i totally don't understand your comment, @lavalaz. First you write "Ideal" and then you bash the idea, I'm confused.

[lavalaz]

Scenario #2: A user returns to ISIS after a long hiatus, expecting that there are changes to how ISIS works and that the documentation will make it easy to find the specific changes that affect their work.

Story: I am a researcher who returns to a project after a 2.5 year hiatus and am planning to process new data using the isis-heavy pipeline I used back then. But before I start I do a quick search on the isis documentation to see if anything has changed with the applications I am using. I find that an obscure issue (maybe a bad interaction between special pixels and a calibration routine) has been corrected, but I need to set a parameter to a different value to activate this fix. I make the small change to my old pipeline, reprocess the old and new data. Brilliant publication follows shortly with glowing acknowledgement to isis.

Note: What this story/scenario is meant to emphasize is the need not just for thorough documentation but also the ability to "discover" or search for the most relevant pieces, like new parameters available within a specific application.

[lavalaz]

@michaelaye , I am writing a story of what I would like in an ideal world, but I also know that what I am asking for is unreasonable.

[ladoramkershner]

I am an internal user and work on creating products of the moon and mars. I almost exclusively use the internal conda environments and rely on a cluster to handle my processing. I use 4.0.1 because I have not needed to update as I work with very stable/mature functions (jigsaw, automos, etc.) and I subscribe to an "if it ain't broke (and there are no enhancements), don't fix it" mentality. As for release stability needs, taken from the ISO 25010 standars linked in the original post:

Category	Need it	Would Be Nice	Neutral
Functional Suitability	Completeness, Correctness, Appropriateness
Performance Efficiency		Time Behavior, Resource	Capacity
Compatibility	Interoperability		Coexistence
Usability	Appropriateness recognizability, Learnability, Operability	User error protection, User interface aesthetics	Accessibility
Reliability	Maturity, Availability	Fault Tolerance, Recoverability
Security
Maintainability
Portability

Category	Explanation
Functional Suitability	I need ISIS to function and do all the things I need. If anything in particular is missing I feel comfortable submitting a ticket to get it addressed.
Performance Efficiency	I have access to a cluster with a lot of resources where I can parallelize a lot of functions/processes (spiceinit, footprintinit, cam2map, etc.). So, I am not terribly concerned with Time Behavior and Resource for these functions. However, I do care about these categories when it comes to jigsaw, automos, and other functions that cannot be parallelized.
Compatibility	When processing data, I use multiple services/languages, so it is important to me that ISIS can exchange data with these services and/or be interpreted in other languages. On the other hand, the cluster allows me to not worry about ISIS ‘playing nice’ with other services, so I don’t really care about that.
Usability	I am internal and went through the old ISIS Workshops for my training, but a lot of what I have learned and some of the most useful information I have gotten has been from coworkers who are much more familiar with ISIS, it use, and its interfacing with different instruments. I am not sure how proficient I would be if I did not work with such experienced people. I think this lack of explicit documentation/training materials is a big hinderance for ISIS getting used more in academia (and I assume other places as well). Therefore, I think appropriateness recognizability and learnability are needed and one of our biggest improvement areas.   There are so many different uses for ISIS that is it difficult to set up proper errors for every case or mission. However, I think that general errors should be in place to ensure people are not using functions outside of their intended use. I personally think this is done pretty well in the current code base (aka I have not experienced any troubles). Also, I really enjoy the explicit nature of using the command line for processing in ISIS, so aesthetics (aka guis) are not very important to me. However, I do think the qisis should be aesthetic in the sense it should be easy to interpret thumbnail symbols, so it is easy to navigate quickly.   ISIS is a fairly specialized software that is used for scientist/cartographers/other technical fields studying inner planets. Since this user base has presumably higher than average accessibility to everyone is not as important as the other usability sub-categories
Reliability	I need ISIS to work when I need it and it should work reliably/repeatably. I do not think that needs explanation but let me know if you would like one. I create pipelines/workflows for my data processing, so I would rather a loud failure instead of ISIS limping along if something goes out or accidentally munging the data in some way when trying to recover data.

I am skipping the last 3 categories, I feel like those are not applicable to a products user story. But I also I wanted to keep them in the table in case anyone wanted to copy/reuse!

[rheyd]

I am a ground systems manager and data processing pipeline developer. My story is very similar to that of rbeyer's. I work on a mature spacecraft mission where the majority of processing needs are met, but development of new data products sill occur. I am also involved in another mission that is maturing, but is still under under heavy development.

In a mature mission context, reliability and consistency are most important to us. Changes that necessitate updating parameters in our existing code can be tricky to find, even when such changes are well documented within a new Isis release. Testing new versions of Isis can be very time consuming, especially as our staffing has been reduced. Frequent releases make it difficult for us to keep current and our solution to this is to not try to keep up, but be strategic about when we upgrade. Our current policy is to upgrade at least once a year, but no more than twice. We have chosen this route instead of "freezing" our GDS as we still have on going development that needs Isis support and falling behind too far means much more work than keeping up.

In the context of a developing mission, we need to be able to work closely with the Isis team to be sure they are aware of our developing needs and be able to have rapid updates to the Isis applications we are using to develop our processing system. Bugs are frequent and need to be addressed quickly to allow the development process to progress.

[victoronline]

As a developer for a mature, ongoing mission, I need to develop against a version of ISIS that is stable and would not introduce new issues but also allows us to remain at that version until we can upgrade to a later version as mission time allows.

One hard requirement is that I need to remain at a stable software version but have up-to-the-minute data. Our mission is actively producing data but can't match the release cadence. We need to be able to work off a set software version but still have the latest version of data.

Ideally, I'd like to keep up with major versions, but the release cadence is too fast. Upgrading consists of updating our environment to make the version available to our our science team to test. Since the adoption of Anaconda, which is geared more for creating an individual environments, our network environment requires some effort. Adopting a new release takes mission time. Once this release is available, it must be validated with our pipelines and production processes. This requires a good amount of effort and coordination. The current cadence does not allow us to do this for every version released, thus creating a situation where in order to adopt bug fixes, we have to upgrade to a more current version. This requires testing and validation from not only our developers, but also our science team. Having to reproduce products that do not contain errors attributed to bugs or breaks created by bug fixes, is very time consuming and requires a lot effort coordination. We have over 10 years of data. If we are unable to move to a release that contains a critical bug fix, I have to develop and implement a local bug fix or release enhancement as a work-around until we can upgrade to the release containing the fix. 

Realistically, if we had a release schedule, we could synchronize our development, testing, and validation schedule proportionally against major versions, with skipping a set number of major versions at a time. 

Ideally, we would like the option to adopt releases containing only bug fixes, or features, but not containing both

[KrisBecker]

I am an image data analyst that often evaluates interactions between ISIS, DTMs and navigation data on small, irregular bodies. I rely heavily on output data produced by applications such as instrument import programs (i.e., label content), spiceinit, isisminer, camstats, footprintint, cnetcheck and jigsaw. Stability, consistency and content with regard to units and order are critical to my work. At times, its difficult to determine presence of data and units from jigsaw due to different input parameters.

One of the most important aspects of our analysis work is comparisons and validation of jigsaw bundle adjustments to digital shape kernel (DSK) models. We use high resolution DSKs as ground truth. Comparisons of jigsaw radius solutions at adjusted latitude/longitudes are compared directly with radii in the DSK at the same locations. This is only effective when jigsaw units are in body fixed coordinates.

I am not too concerned with control network formats as I find it sufficient to associate comprehensive databases/CSVs produced from isisminer and output files from cnetcheck and jigsaw in production cycles.

Changes in application parameters, output file formats/content or units with new ISIS releases can cause disruptions in workflows and make comparisons/verifications between releases difficult. Thorough documentation related to these features is also a must.

I realize that it may be too much to assume changes won’t occur between major ISIS releases, so I would try to make my analysis immune to changes as much as possible (e.g, relying on consistent column header names and units) by anticipating reordering in or new additions of data (columns). isisminer is not a problem – jigsaw is as it doesn’t currently have a reliable single column header line naming scheme in most if its output files.

[jessemapel]

I am closing this out as we are moving onto decision making. If anyone has further comments please add them here and re-open.