CASEI assessed for potential refactoring

[heidimok]

Summary of Issue

DevSeed to better illustrate to ESDS what an overhaul to CASEI (as it is today) would look like given where the system is at.

Technical proposal should include:

• [x] #539
• [x] #543
• [ ] Relevant tests conducted around data volume - scalability
• [ ] Consolidated pro/con list with recommendation shared by developers on ADMG
• [ ] Projection of time and effort (E.g. how many sprints or PIs) would be required to act on the recommendations

Expected Outcome: ADMG can have the information they need to make a decision about our technical roadmap

---

More Context

• The CASEI project started in 2020 with the initial target date for handoff in Jan 2021. We had a full release of the interface in 2021 but it was considered “beta”. In July 2023, we will have a non-beta public release with now a majority (65%) of known NASA airborne and field campaigns represented across the database and a unique range of search/browse functionality built into CASEI and its API. Additional campaigns, platforms, instruments, and data products are still being curated. Meanwhile, transition talks between ADMG and ESDS are just getting started and the timeline to official handoff is still TBD.
• However, periodically throughout FY23, challenges to the maintenance of CASEI have come up among developers. Due to other ongoing priorities like getting to a non-beta release, we’ve mostly pushed through on features rather than technical debt.
• During a recent retro discussion about why we had MI deployment failures, Carson said that technical debt issues were brought to the service during our recent backend deployment of DOI changes in staging. It was the first time in a while we had a whole system update on changes that touched the deepest level of the system. Carson said that the system is more complicated than it needs to be. The draft system can make it easy for stuff to go wrong. Carson said that we should have to look at that next PI because at one point in our history, the draft when you did updates only carried the diffs, but now they carry the entire object including the diff. You might not see anything go wrong until you update the whole system.
• Main Concern for the Future: If we carry on with CASEI as it is, it may be difficult for developers to make fixes and add new features without unintentionally breaking the system.
  • It’s hard to reason with what’s going on in the system. It’s easy to get lost about how a request is going point A to B.
  • The level of abstraction is really high. One anecdote was that it took a developer who knew the system 3 days to figure out how something was rendered. Because it’s hard to reason about, it’s hard to break. It’s the downstream effect of changes.
  • We’re in agreement that future maintainability of CASEI is going to be a steep onboarding experience for a developer, will require a lot of willpower and time, plus a good understanding of django. It’s impossible to fully document everything.
  • The system will definitely need to run for 5+ years after we hand it off.
• How did we get here? Why is our system more complicated than it needs to be? Just one explanation: We made a decision really early on in the project to build the draft system by hand. But because there are certain elements in the database that are related to each other (e.g. instrument and platform on the same flight), there’s an automated thing called cascading that happens (usually baked into the tools we use). When cascades and migrations happen in the data model, they don’t happen in the change objects. Due to the custom build the drafts don’t get handled automatically.
• Major Risks of Change
  • Major refactoring or rewriting takes a long time
  • It’s not always greener on the other side. Even if you know the current faults of a system and can imagine a more perfect system that fits the current problem, you can end up in the same state as when we started. We know more about the stable state, but there is risk in changing everything as well. Unless we go through a very thorough planning phase, big refactors can go off the rails very quickly.
  • We are at point in the project where our focus is closing out and transitioning to ESDS. If we do not expect to develop much on the backend there is a question of whether it’s worth the investment. If we are at the point of handover at end of FY24, does it make sense to invest in a “full house remodel”.

[heidimok]

Closing this now in favor of this epic: https://github.com/NASA-IMPACT/admg-backend/issues/546