The problem is that every time we generate a new derived column, we take last id eg: d3 and then we increment it resulting in d4.
1- Let's image we do the following transformation sequence: derive, derive, delete-last-column-derived, derive, delete-last-column-derived, derive we end with following columns d1, d2, d3 but we have 6 transformations
2- Now we make a viz that uses column d2
3- try to update the dataset
internally lumen tries following:
1- grab raw flow data ........................... OK
2- apply derive transformtion => d1 ..... OK
3- apply derive transformtion => d2 .... OK
3- delete last derive transformtion => d2 .... NOK thus d2 is used in viz
Solution or next step
Improve way of ensuring data-consistency on delete-columns
Context
When we rerun transformations (update & undo) then lumen checks transformation constraints In case of delete-column https://github.com/akvo/akvo-lumen/blob/master/backend/src/akvo/lumen/lib/transformation/delete_column.clj#L41-L45 we query the db looking for visualisations that contain this columnName to ensure data consistency.
Problem or idea
The problem is that every time we generate a new derived column, we take last id eg:
d3and then we increment it resulting ind4. 1- Let's image we do the following transformation sequence: derive, derive, delete-last-column-derived, derive, delete-last-column-derived, derive we end with following columns d1, d2, d3 but we have 6 transformations 2- Now we make a viz that uses column d2 3- try to update the datasetinternally lumen tries following: 1- grab raw flow data ........................... OK 2- apply derive transformtion => d1 ..... OK 3- apply derive transformtion => d2 .... OK 3- delete last derive transformtion => d2 .... NOK thus d2 is used in viz
Solution or next step
Improve way of ensuring data-consistency on delete-columns