Open staadecker opened 1 year ago
Hello @PatyHidalgo @mfripp!
I spent most of my weekend exploring our options for joining the wecc
branch with the master
branch. There are several concerns that come to mind and make me think we might need to reconsider the approach. For context, I made both branches as similar as possible (i.e. reformatting both branches with black so that formatting differences don't cause merge conflicts, see above). Nonetheless, there are 87 files with merge conflicts that need to be resolved manually and 371 more files that would be added or changed due to a merge (some of which will require integration into the upstream switch
system). My concerns are as follows:
master
into wecc
will change the wecc
code. Since I'm no longer an active user of switch-wecc, I would need support from your current students @PatyHidalgo to test these changes and make sure they don't break anything. Otherwise, it is very likely the changes will break the wecc
version and your team won't migrate from the REAM repo to this repo. What is the bandwidth of your current team and what capabilities do they have in terms of being actively involved in fixing bugs and migrating their projects to new versions of SWITCH?wecc
into master
(see list above), we would need to keep a good pace on merge requests if we want to ever get this done. I see there are already several pending merge requests that are in need of attention. How busy are you at the moment @mfripp and are you able to dedicate more time weekly to get the changes merged?wecc
and master
versions have approached things differently. There is significant new code required to merge the branches and the experience for your team @PatyHidalgo will need to change. For example, switch get_inputs
would have to change to switch wecc get_inputs
because we don't want all switch users to worry about get_inputs
which is wecc
specific. In this example, the change isn't complicated but it illustrates how some code will need to be re-written and I expect some other cases with be significantly more complicated. How would we approach resolving differences in approaches and how can we make sure your team @PatyHidalgo is constantly adapting to the new changes?I see a few options:
wecc
branch that @mfripp you really want in the main repository (with your support to get it tested and merged quickly before merge conflicts re-appear). The switch
and switch-wecc
would still be separate projects but at least the switch
community could benefit from a few cool features.Looking forward to figuring this out together! Working on open-source projects with limited resources is always a challenge but let's do the best we can. I would be very happy if the switch community can benefit from the switch-wecc features and if both teams can work on a unified project to make the benefits accessible to the entire community. I am sad that I can't lead this effort (I really would like to but I'm trying to make the smart decision and not overcommit myself as I usually do) but I will do what I can to help as much as possible!
One more thing: I've thought of a good example to illustrate the challenge in resolving the differences between the switch-wecc
and switch
versions.
switch-wecc
has scaled some of the core model's equations to improve the numerical properties and performance of its model. However, this change might be detrimental to another team depending on the exact numerical properties of their model. To merge this change their are 3 options:
switch-wecc
scaling but this might worsen the performance for other switch
users.switch
scaling, this will very likely worsen the performance for the switch-wecc
model and REAM team.
This issue will serve as a central point to discuss merging the issues from the SWITCH WECC repo. As of now the plan is as follows.
Action plan
wecc
branch in this repo.wecc
specific files into awecc
package.master-with-black-formatted-history
andwecc-with-black-formatted-history
which are themaster
andwecc
branches rewritten but with their history modified such that every commit is formatted according toblack
. This reduces merge conflicts in the next step (see notes on how this was done below).master
intowecc
wecc
branch into the main branch as small features. These merge requests will have to be merged more or less in order. See chart below.WECC features to include into
master
switch drop
tool (status: not started)wecc
folder for others to see / use (status: not started)switch compare
(status: not started)master
features to include in WECCdimen
in set definitionConflicting changes
ordered=False
inwecc
butunique_list
inmaster
wecc
specific examples e.g.ca_policies
orstochastic
exampleconfig.yaml
approach to generating scenariosNotes
Process for creating
master-with-black-formatted-history
andwecc-with-black-formatted-history
This is achieved with the following commands in Powershell.
Run
git merge-base wecc master
to find the commit where the two branches first diverged.Checkout that commit on a new branch called
common-wecc-parent
.Reformat the code with
black --fast -t py310 --extend-exclude 'switch_wecc/wecc' ./switch_model
and make a new commit.Checkout
master
on switch to a new branchmaster-with-black-formatted-history
.Run
git rebase --rebase-merges --exec "black --fast -t py310 --extend-exclude 'switch_wecc/wecc' ./switch_model; git commit -a --amend --allow-empty" common-wecc-parent
a. The--rebase-merges
ensures we keep the topography of the merges b. The--exec "black --fast -t py310 --extend-exclude 'switch_wecc/wecc' ./switch_model; git commit -a --amend --allow-empty"
means that after every commit, we amend the commit such that the switch_model files are reformatted.This will generate significant merge conflicts. For each one run a.
git restore --theirs -s REBASE_HEAD .
to update the local files to the commit that is being applied. b.black --fast -t py310 --extend-exclude 'switch_wecc/wecc' ./switch_model
to reformat the files. c.git add .
to mark any conflicts as resolved. d.git rebase --continue
to continue with the rebase. To avoid the popup to edit the commit message see instructions here. To make these commands automatically run up to 100 times in a loop, in Powershell simply run1..10 | % {git restore --theirs -s REBASE_HEAD . ; black --fast -t py310 --extend-exclude 'switch_wecc/wecc' ./switch_model ; git add . ; git rebase --continue}
Check that there are no differences between
master-with-black-formatted-history
andmaster
(you might need to run black onmaster
)Redo step 4 to 7 but for the
wecc
branch.