Zejnilovic
commented
5 years ago Copied from #194
Describe the bug
Currently, every little change I do in Menas creates a new version. I add Conformance Rules, it creates a version, I upload a schema, it adds a version, etc. This then leads to a piling of unusable references with no ability to tag the versions. Which leads to a huge decrease in usability.
Imagine. I have a Dataset with more than 20 conformance rules. And the only usable version is version number 2X (let's say 23). I then notice something changed. What do I do? I change what is needed. Next usable version is 35. I have to have a separate wiki page with a list of usable versions to track them.
Now I want to see where it is used and I can scroll and scroll.
I hope this is enough to show what I mean.
Expected behaviour
I would expect some flag "work in progress" and create a version only when I hit some publish button.
With the new UI, all changes even micro changes are incurring multiple versions, this is good and bad as we have a full roll back on any change to a very granular point. The proposed idea, is to have a separate version system for tracking changes in Menas (when creating and updating definition) compared to version of the dataset we run when doing spark-submit, we will have too many versions due to micro-changes that Menas is going to introduce