search

trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.19k stars 565 forks source link

Initial transition to the 'develop'/'master' workflow #370

Closed bartlettroscoe closed 7 years ago

bartlettroscoe commented 8 years ago

Next Action Status:

Brent is manually updating 'master' from 'develop' about once a day using fast-forward merges. Next: Automate the update from 'develop' to 'master' based on new post-push CI build (see #482) ...

CC: @jwillenbring, @bmpersc, @maherou, @rppawlo

Blocked By: #158, #410

Blocking: #380

Description:

This story is to transition Trilinos to the 'develop'/'master' branch workflow which is described in great detail here.

In summary, developers will commit directly to the ‘develop’ branch and then a --no-ff merge will be performed to merge from develop to master when selected builds and package tests all pass (hopefully once every day). Initially, we will just have a single CI build that is run using the checkin-test.py script (based on the SEMS Dev Env, see #158) and if that build passes, the merge to 'master' will occur. Longer term, we will use the new CDash API to query to decide if to update 'master' or not.

Definition of Done:

Tasks:

  1. Write up instructions on how to do the 'develop' branch workflow ... see here [Done]
  2. Decide on timeline for transition ... see below [Done]
  3. Write up instructions on how to move commits from existing local 'master' branch to local 'develop' branch ... see below and here [Done]
  4. Create ‘develop’ branch off of ‘master’ and push to GitHub repo ... see below [Done]
  5. Tuesday, 5/31/2016: Send out email announcing the transition and point to the wiki documentation (i.e. do full initial git setup, get on local tracking 'develop' branch and transitioning local commits from 'master' to 'develop') ... see below [Done]
  6. Discuss impact on 'develop'/'master' workflow and possible back-outs of merges to 'master' on customer projects like 'Drekar' and others ... They will just initially pull from 'develop' instead of 'master' (see below) [Done]
  7. Wednesday 6/8/2016: Do the transition (i..e "flip the switch"):
    • Send out email to trilinos-developers reminding of the the transition (and pointing to documentation materials) ... see below [Done]
    • Send out email to trilinos-developers to mark the start of the transition (can't push to 'master'). [Done]
    • Block who can directly push to ‘master’ on github repo (allow just a few people to push to 'master' initially). [Done]
    • Update the github ‘develop’ to current tip of github ‘master’. [Done]
    • Open up pushes to the 'develop' branch. [Done]
    • Send out email to trilinos-developers announcing that the transition is complete (again pointing to documentation and provide list of people they can ask for help). ... see below [Done]
  8. Switch over automated CI and Nightly testing to pull from 'develop' instead of 'master' (see below). [Done]
  9. For first few days, just do manual updates from 'develop' to 'master'. ... IN PROGRESS (this has been going on for nearly 6 months!) ...
  10. Replace manual updates of 'master' with cron/Jenkins job that runs the standard CI build (#482) using standard SEMS Dev Env (#158) and updated set of PT packages and TPLs (#410) as gate to update from 'develop' to 'master' ... Deferred to #982
mhoemmen commented 8 years ago

What are the criteria for updating master? I would specify them like this:

  1. Subset of packages that must build and pass tests, with
  2. Subset of supported compilers, and with
  3. As many TPLs enabled as people can reasonably set up.

The first ensures no surprises. If you work on a package in that subset, and your tests aren't passing, you know you're on the hook. The second avoids the "works for me" syndrome. I specified the third because (a) it's too hard to test all possible TPL enabled combinations, and (b) usually, enabling a TPL enables more code, so test coverage tends to increase if more TPLs are enabled.

bartlettroscoe commented 8 years ago

What are the criteria for updating master?

@mhoemmen, initially, as stated above in the "Description" field (i.e. the first comment):

The question of how to expand this to other builds and compilers and TPLs, etc. will be addressed in the follow-on story #380. Hopefully #380 will help to address most of your questions. Please read that over and then let's discuss this at the next (or a near term) Trilinos Leaders meeting.

But first we just need to make the switch to the 'develop'/'master' branch workflow and then we can worry about how to do more testing before updating the 'master' branch.

mhoemmen commented 8 years ago

But first we just need to make the switch to the 'develop'/'master' branch workflow and then we can worry about how to do more testing before updating the 'master' branch.

I'm OK with that.

jwillenbring commented 8 years ago

Just a few comments:

1) As I mentioned to Ross on the phone, I question if CDash is the right tool to decide if the develop to master promotion should take place. This conversation probably shouldn't take place in this specific ticket, but I wanted to mention it.

2) I would add 4a) Announce timeline to developers and important users.

3) I would add as a note for #9 that the checkin-test.py script needs to run a set of builds/tests that have 100% passing status. We can grow the set of tests that run, but w can't start with any failures on master when we create develop.

Note on costs:

Although not directly attributable to this model, we would concurrently (with moving to the 2 branch model) be enforcing additional stability checks on the code in the repo. A strict commitment to keeping things 100% passing is more of change in some respects than the technical git details associated with this process. This might sound obvious, but I think we need to focus on those challenges more than what it takes to check out develop before committing.

bartlettroscoe commented 8 years ago

I added the section Transitioning local changes to the 'develop' branch to the 'develop'/'master' workflow page.

Next I will actually create the 'develop' branch in the github repo so that people can create their local 'develop' branch. But I will lock down who can push to it to avoid problems.

bartlettroscoe commented 8 years ago

I have just now created the branch 'develop' off of 'master' and pushed it to the main github repo:

https://github.com/trilinos/Trilinos/settings/branches/develop

I have restricted the list of people who can push, for now, to just @jwillenbring, @bmpersc and myself. After we make the switch, we will open up pushing to the 'develop' branch and then lock down who can push to the 'master' branch.

@jwillenbring and/or @bmpersc, can you please review the page 'develop' 'master' workflow? Now that I have created the shared 'develop' branch in github, you can copy and paste the command and make sure that there are not typos.

Next I will post the time table for making the switch we decided on Tuesday at the Trilinos Leaders Meeting and send out an email about this.

bartlettroscoe commented 8 years ago

At the Trilinos Leaders Meeting on 5/25/2016, we decided on the following timeline:

We should send out an email reminder a few days before 6/8/2016 reminding developers about the switch.

bartlettroscoe commented 8 years ago

At the Trilinos Leaders Meeting on 5/31/2016, there was a lot of discussion on this topic. Some of the developers had a lot of questions and concerns about what this would mean. One thing that came out of this is that, no matter what the update process, that it be made transparent and visible to the developers. Initially I think this means sending out an email to the trilinos-developers mail list when the 'master' branch gets updated from the 'develop' branch. We can do that with the initial implementation with the checkin-test.py script in that it will send out an email when it updates the branch. The CTest/CDash CI build should be made to exactly match this build so that developers will get an email when and update occurs (or fails).

dridzal commented 8 years ago

Thanks Ross, this will be very helpful initially. We should also start thinking about the list of packages that must pass the tests before a merge with 'master' is performed. Otherwise, there could be a lot of controversy come June 9. Specifically: Which packages? Which tests/compilers? What is the timeline for action related to failing tests (e.g., before a package is removed from the above list)? Where is all this documented?

From: Roscoe A. Bartlett notifications@github.com Sent: Tuesday, May 31, 2016 14:56 To: trilinos/Trilinos Subject: [EXTERNAL] Re: [trilinos/Trilinos] Initial transition to the 'develop'/'master' workflow (#370)

At the Trilinos Leaders Meeting on 5/31/2016https://docs.google.com/document/d/1cTbUH5NlJlrlZfjXPdUYjhBcxmE2fiOINfs0UPVV-w0, there was a lot of discussion on this topic. Some of the developers had a lot of questions and concerns about what this would mean. One thing that came out of this is that, no matter what the update process, that it be made transparent and visible to the developers. Initially I think this means sending out an email to the trilinos-developers mail list when the 'master' branch gets updated from the 'develop' branch. We can do that with the initial implementation with the checkin-test.py script in that it will send out an email when it updates the branch. The CTest/CDash CI build should be made to exactly match this build so that developers will get an email when and update occurs (or fails).

You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/trilinos/Trilinos/issues/370#issuecomment-222817882, or mute the threadhttps://github.com/notifications/unsubscribe/APE1CvVqGh1rna6fAeqPuG7rcwsHY3qoks5qHKCCgaJpZM4IiVCT.

jwillenbring commented 8 years ago

@bartlettroscoe How would you like me to submit a few documentation suggestions? Should I just add them to the wiki, make a list here, etc?

bartlettroscoe commented 8 years ago

How would you like me to submit a few documentation suggestions? Should I just add them to the wiki, make a list here, etc?

@jwillenbring,

If the changes are obvious, then I would just go ahead and make the changes in wiki. Just ttry to create logically consistent commits and then I will be able to clone Trilinos GitHub Wiki (git@github.com:trilinos/Trilinos.wiki.git) and review the changes there.

If the changes are not obvious, another option is to create a TrilinosWiki GitHub repo that is actually a clone of the trilinos/Trilinos.wiki.git repo and then we could use pull requests on that repo to propose and review changes. I just demonstrated this at:

https://github.com/bartlettroscoe/TrilinosWiki

For example, see:

https://github.com/bartlettroscoe/TrilinosWiki/blob/master/VC-|-'develop'-'master'-workflow.md

Why not use GitHub workflows to collaboratively make changes to the Trilinos Wiki? Should we create a TrilinosWiki repo under github.com/trilinos/ for this purpose? Do you want to give that a try?

bartlettroscoe commented 8 years ago

I added the new story #410 to specifically address the adjusted set of Primary Tested packages that will be used with the checkin-test.py script for the initial implementation of the cron job that will update 'master' from 'develop'.

jwillenbring commented 8 years ago

So the idea would be to then just merge the changes from the wiki repo back to the actual wiki by cloning it? I saw you can't actually just do pull requests against the wiki.

bartlettroscoe commented 8 years ago

So the idea would be to then just merge the changes from the wiki repo back to the actual wiki by cloning it? I saw you can't actually just do pull requests against the wiki.

You use an intermediate local git repo to manage the chagnes. You make the changes there and the push to a branch in your github clone/fork. For example, I did:

$ git clone git@github.com:trilinos/Trilinos.wiki.git
$ cd Trilinos.wiki
[(master)]$ git remote add bartlettroscoe  git@github.com:bartlettroscoe/TrilinosWiki.git
[(master)]$ git checkout -b fix-some-typos-370
[(fix-some-typos-370)]$ emacs VC-|-'develop'-'master'-workflow.md   # Edit it
[(fix-some-typos-370)]$ git push -u bartlettroscoe fix-some-typos-370 
...
To git@github.com:bartlettroscoe/TrilinosWiki.git
 * [new branch]      fix-some-typos-370 -> fix-some-typos-370
Branch fix-some-typos-370 set up to track remote branch fix-some-typos-370 from bartlettroscoe.

You can then see my updated file at:

https://github.com/bartlettroscoe/TrilinosWiki/blob/fix-some-typos-370/VC-|-'develop'-'master'-workflow.md

I then used that branch to create the PR:

https://github.com/bartlettroscoe/TrilinosWiki/pull/1

Make sense?

So to be super clear, what I am proposing is to create the new GitHub repo:

github.com/trilinos/TrilinosWiki

and push the 'master' branch of the repo trilinos/Trilinos.wiki.git. Then you can use this TrilinosWiki repo to manage large changes to the Trilinos GitHub Wiki. You can edit files live on GitHub on the branch to see your changes in real-time as well. Your choice.

Make sense? If not, let's discuss this briefly and I can demonstrate more.

jwillenbring commented 8 years ago

@bartlettroscoe What you said makes sense, but I thought might be overkill for the level of edits I was suggesting. I created a pull request against your repo you set up. We can look at doing something more advanced in the future for wiki edits, but I thought it would be easy for you to review the changes in a pull request on your clone. Getting the changes back to the Wiki will be similar for you, regardless if the clone is in your github account, or on the Trilinos one.

bartlettroscoe commented 8 years ago

NOTE: I added some text to the main VC Wiki Page that defines the 'develop', 'master' and the release branches and describes there purpose (and who should access what branches). Hopefully this will help align expectations and understanding between Trilinos developers and Trilinos users.

bartlettroscoe commented 8 years ago

I just remembered that as part of this transition, we also need to change the Trilinos 'nightly' branch used in automated testing to be updated from the 'develop' branch instead of the master branch. Also, the Trilinos CI server needs to be updated to pull from develop instead of master. I think the latter is already supported by setting the CMake variable ${PROJECT_NAME}_BRANCH=develop. (Actually, what happened to the Trilinos 'nightly' branch? I don't see 'nightly' listed as one of the active Trilinos branches.

@jwillenbring and @bmpersc, do you see this as being a problem making these changes pretty quickly so that the automated testing processes are testing the 'develop' branch instead of the 'master' branch? I don't have access to those cron jobs so I can't make the needed changes myself.

bartlettroscoe commented 8 years ago

Another issue that just occurred to me is what to do with the extra repos for inserted packages like CTrilinos, ForTrilinos, MOOCHO, Mesquite, Sundance, and extra repos like preCopyrightTrilinos which are listed in the file Trilinos/cmake/ExtraRepositoriesList.cmake. The way TribitsCTestDriverCore.cmake is currently set up, I believe that it will keep pulling from the 'master' branch of these extra repos even through it will be pulling from the Trilinos 'develop' or 'nightly' branches. That is going to become a problem once a non-backward compatible change is made in Trilinos that impacts these other packages.

For the short-term, I think that it is likely okay to just let the automated testing use the 'master' branches on these extra repos and testing them against the Trilinos 'develop' or 'nightly' branch. But longer-term, this is not a good solution. Longer term, I think these repos should also be switched to use a 'develop' branch and a 'nightly' branch and the update process that updates Trilinos from 'develop' to 'master' should also update these other repos from 'develop' to 'master' in their repos as well (that is easy to set up using the gitdist tool). The issue is that we will then have to force the 'develop'/'master' workflow on all of these other repos as well. But I think that is reasonable and the gitdist tool naturally supports that workflow out of the box. I am thinking that all repos that are continuously integrated with Trilinos (i.e. the ones listed in Trilinos/cmake/ExtraRepositoriesList.cmake) should use the 'develop'/'master' workflow and the update of the 'develop' to 'master' branch in each of these extra repos should happen in lock step with the base Trilinos repo's branches.

Another thing to consider is that were are likely not going to block the update of the 'develop' branch to the 'master' branch for Trilinos based on if if any of these extra packages breaks (but we could depending on what packages we pick to include or exclude from our automated builds and/or CDash queries). Also, since the Trilinos packages used by these extra packages tend to maintain backward compatibility, then it will not really matter very much what branches we use for these extra repos in testing and integration. But to smooth changes in backward compatibility in upstream Trilinos packages to these extra packages, I think you have to treat them as one big git repo along with Trilinos (i.e. the gitdist approach), and that means having them have the same 'develop' and 'nightly' branches.

Therefore this is still something to think about and it is worthwhile to set up a Trilinos GitHub issue to address this after this Story is complete.

Any comments on this?

bmpersc commented 8 years ago

nightly was never a branch. It is a separate repository that we only update at a specific time. It should be trivial to update the default branch of the nightly repo. However, that won't automatically change the branches that all the testing is already using. The tests we control I think we can easily fix that by wiping out the workspace and letting the tests clone again. The others we don't control we can alert people to the need to wipe out their workspaces.

bartlettroscoe commented 8 years ago

Below is an email exchange between Roger and I that addresses the task "Discuss impact on 'develop'/'master' workflow and possible back-outs of merges to 'master' on customer projects like 'Drekar' and others.". It sounds like in the short-term, that these projects will just directly pull from 'develop'.

From: Bartlett, Roscoe A Sent: Tuesday, June 07, 2016 1:37 PM To: Pawlowski, Roger P; 'trilinos-framework@software.sandia.gov' Subject: RE: [Trilinos-Framework] Fwd: [EXTERNAL] [Trilinos-developers] reminder about TOMORROW's move to develop - master branch model

Roger,

Shorter-term, I think having Charon, Drekar, and EMPIRE just directly integrate with the Trilinos ‘develop’ branch is likely the right thing to do. But that means that these projects and these other developers will not be benefiting at all from an increasingly more stable Trilinos ‘master’ branch that comes about as the result of the Trilinos ‘develop’/’master’ workflow.

Therefore, longer term, these projects should maintain their own clone of Trilinos (like CASL VERA does) and then changes that require simultaneous pushes can be done directly to Charon’s or Drekar’s or EMPRES’s copy of Trilinos and then shortly after that merged back to the Trilinos ‘develop’ branch. Then these projects would set up an integration process to pull from the more stable Trilinos github ‘master’ branch back into Charon’s, Drekar’s, and EMPIRES’s copy of Trilinos. Since Charon and Drekar are both using TriBITS, that would be fairly easy to set up using the checkin-test.py script (like we do for integrations with CASL VERA). But since EMPIRE is not using TriBITS, they would have to do their own thing (but perhaps we can change that in the future). Such a workflow would take the best advantage of the Trilinos ‘develop’/’master’ workflow and the capabilities of a distributed version control system like git. This type of workflow has worked very well for CASL VERA development.

We should likely talk about this pros and cons of these various and then log our notes in the Trilinos GitHub Issue ticket #370.

-Ross

From: trilinos-framework-bounces@software.sandia.gov [mailto:trilinos-framework-bounces@software.sandia.gov] On Behalf Of Roger Pawlowski Sent: Tuesday, June 07, 2016 1:05 PM To: 'trilinos-framework@software.sandia.gov' Subject: [Trilinos-Framework] Fwd: [EXTERNAL] [Trilinos-developers] reminder about TOMORROW's move to develop - master branch model

So what is the path forward for multi-repo projects that rely on simultaneous changes to multiple repos in a single push. For example if I make changes to panzer that require simultaneous changes to Charon, Drekar and EMPIRE (happens quite frequently) I think that the easiest thing to do is change the testing of all application repos to use develop. Otherwise I need to screw with tons of ifdefs - these kinds of invasive changes tend to touch quite a bit of code.

Roger

bartlettroscoe commented 8 years ago

nightly was never a branch. It is a separate repository that we only update at a specific time. It should be trivial to update the default branch of the nightly repo.

Right, I think I remember that now. I see that in the file Trilinos/cmake/ctest/TrilinosCTestDriverCore.cmake in:

SET(Trilinos_REPOSITORY_LOCATION_NIGHTLY_DEFAULT
  "software.sandia.gov:/space/git/nightly/Trilinos")

Therefore, we just need to update the cron job that updates this repo to update from the Trilinos github 'develop' branch instead of the 'master' branch. Where is the cron job that does that?

Down the line, I think we should change to use a 'nightly' branch instead of a 'nightly' repo. That way we can set up testing jobs for Trilinos that can't pull from ssg.

However, that won't automatically change the branches that all the testing is already using. The tests we control I think we can easily fix that by wiping out the workspace and letting the tests clone again. The others we don't control we can alert people to the need to wipe out their workspaces.

Given that the 'nightly' repo on ssg is being used, I think this takes care of itself for now, right?

Therefore, I think we just need to get the Trilinos CI server running on sadl30906.srn.sandia.gov to test the 'develop' branch instead of the 'master' branch. That should be as easy as setting the branch to 'develop' as described above.

bartlettroscoe commented 8 years ago

The email I sent out on May 31 is shown below.

I think the email I send out tomorrow is just going to be a forwarding of the below email with updated details.

From: Bartlett, Roscoe A Sent: Wednesday, June 01, 2016 9:15 AM To: trilinos-developers@trilinos.org Cc: Trilinos Framework Subject: Transition to 'develop'/'master' branch worfkflow on June 8 Importance: High

Hello Trilinos Developers,

Trilinos is transitioning to the ‘develop’/’master’ branch workflow. Starting Wednesday, June 8, average Trilinos developers will no longer be able to push changes directly the Trilinos github ‘master’ branch but instead will need to push to the ‘develop’ branch. The ‘master’ branch will then be updated form the ‘develop’ branch to maintain stability of the ‘master’ branch. This was discussed at the Trilinos Spring Developers Meeting on May 12 (and the prior TSDM and TUG meetings in 2015) and the last several Trilinos Leaders Meetings.

The details on how this will impact your development work and how you can adjust are given at:

https://github.com/trilinos/Trilinos/wiki/VC-|-'develop'-'master'-workflow

The detailed initial transition plan and related stories are given in the list of “tasks” in:

https://github.com/trilinos/Trilinos/issues/370

If you have general questions, concerns, or comments about this, please provide them as comments in Issue #370. (However, if you have Sandia-specific issues with this, then send email to trilinos-framework and/or the sandia-trilinos-developers email lists instead of updating #370 or other Issue tickets on GitHub.)

-Ross

P.S. If you did not attend the Trilinos Spring Developers Meeting or any of the recent Trilinos Leaders Meetings, then you will likely have a lot of questions about this change. To understand the motivations and basic mechanics for this workflow, please see the section “Addition of a ‘develop’ branch” at:

https://docs.google.com/document/d/1uVQYI2cmNx09fDkHDA136yqDTqayhxqfvjFiuUue7wo/edit#heading=h.u2ougk1wk7ph

The details for Trilinos are tracked in Trilinos GitHub #370 and it’s “Blocking” and “Blocked By” Issues. If you are interested in being part of those conversations, then please add a comment to these various Issue tickets.

bartlettroscoe commented 8 years ago

I think we have worked out the details on the switch of the nightly and CI testing to use the 'develop' branch (see below).

I will ask Mike H. for an account on his Linux machine sadl30906.srn.sandia.gov so that I can test out switching the CI server to use the 'develop' branch instead of the 'master' branch.

From: Bartlett, Roscoe A Sent: Tuesday, June 07, 2016 5:09 PM To: Perschbacher, Brent M; Willenbring, James M Subject: RE: updating nightly repo for develop master change

Brent,

Thanks. I found the same thing (and just sent it to you).

So the process to update the testing will be as follows?

  1. Make the official switch from ‘master’ to ‘develop’ as per task 7 in #370 [Ross]
  2. Change the default branch in the ‘nightly’ repo to ‘develop’ [Brent]
  3. Kill all of the automated testing servers, blow away their local Trilinos clones, and send emails to trilinos-developers telling them to do the same for their CTest/CDash drivers as well
  4. Let Jenkins and people’s various cron jobs restart the builds which will clone the ‘nightly’ repo from scratch but now with ‘develop’ as the default branch
  5. Change the branch for the CI server from ‘master’ to ‘develop’ (since it will be pulling from github)

If I can get an account on the CI test machine, I think that I can test the change for the CI server and test it to make sure that it will work.

Is this a plan?

-Ross

From: Perschbacher, Brent M Sent: Tuesday, June 07, 2016 4:59 PM To: Bartlett, Roscoe A; Willenbring, James M Subject: updating nightly repo for develop master change

Ross, It isn’t checkout that is needed to change the default branch, but there is a simple command to do it. It requires access to the repository, but we have that. The command "git symbolic-ref HEAD refs/heads/master" would set the default branch to master. I’m sure that the last time I did this I was cloning from a repo with a working tree since checkout also modifies the HEAD which would affect any clone.

I think we have what we need to change the default branch. Let me know when we are ready to switch tomorrow.

Brent

bartlettroscoe commented 8 years ago

So to update CI server, we could also just change ??? to make it checkout a tracking 'develop' branch. But setting Triilnos_BRANCH=develop should also do the trick and should be perfectly safe.

It is good to know that once your local repo is cloned and is on a tracking branch, CTest will not mess with that. I will make sure that TriBITS' handling of extra repos has the same behavior. But if you set ${PROJECT_NAME}_BRANCH

-----Original Message----- From: Brad King [mailto:brad.king@kitware.com] Sent: Wednesday, June 08, 2016 8:03 AM To: Bartlett, Roscoe A; Betsy McPhail; Zack Galbreath; Bill Hoffman Subject: Re: [EXTERNAL] Re: FW: Get ctest -S script to pull from git branch other than 'master'?

On 06/08/2016 09:40 AM, Bartlett, Roscoe A wrote:

Can we update the CTest documentation to provide this information?

Done:

Help: Document ctest_update branch following behavior https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=558e4d1e

However, I think that I have observed that CTest will also blow away any local uncommitted changes before it does the 'git pull'. Is that true?

Yes:

Help: Document CTest Git fetch-and-reset behavior https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=1b18180e

The idea is that any source tree maintained by ctest_update is meant to test the version in the followed branch.

So what CTest command actually does the clone if the repo? I thought that ctest_update() will do the clone if it is not already been made. Does it just use the default branch in the remote repo?

The ctest_start() command does this with CTEST_CHECKOUT_COMMAND:

https://cmake.org/cmake/help/v3.6/command/ctest_start.html

https://cmake.org/cmake/help/v3.6/variable/CTEST_CHECKOUT_COMMAND.ht ml

It is up to the CTest script to set CTEST_CHECKOUT_COMMAND if it wants an initial checkout to be done.

-Brad

bartlettroscoe commented 8 years ago

Jim did a review and edit of the 'develop'/'master' workflow wiki page and I reviewed and pushed it. Thanks, Jim, it looks very nice!

gsjaardema commented 8 years ago

It looks like develop is behind master at this time. Is someone going to merge the recent master commits into develop so we can start at the same point?

bartlettroscoe commented 8 years ago

It looks like develop is behind master at this time. Is someone going to merge the recent master commits into develop so we can start at the same point?

Yes, see "Update the github ‘develop’ to current tip of github ‘master’" above in list of tasks for the transition today.

bartlettroscoe commented 8 years ago

I send out the following email as per plan.

From: Bartlett, Roscoe A Sent: Wednesday, June 08, 2016 9:10 AM To: 'trilinos-developers@trilinos.org' Cc: 'Trilinos Framework' Subject: RE: Transition to 'develop'/'master' branch workflow on June 8

Hello Trilinos Developers,

We will be switching to the ‘develop’/’master workflow today starting at about 12:30 PM MDT. I will send out an email when the transition starts and one when the transition is complete.

The wiki page on this has been updated to make it simpler/better (Thanks Jim!):

https://github.com/trilinos/Trilinos/wiki/VC-|-'develop'-'master'-workflow

NOTE: Please do not rush and push to ‘master’ before this change! You can easily transfer your local work to the ‘develop’ branch after the switch. See:

https://github.com/trilinos/Trilinos/wiki/VC-%7C-'develop'-'master'-workflow#move_to_develop

Cheers,

-Ross

bartlettroscoe commented 8 years ago

The initial transition to the 'develop' branch is complete (see details below).

Next:

DETAILED NOTES:

Transition to the 'develop' branch:

a) Send out email to trilinos-developers to mark the start of the transition (can't push to 'master')

I send out the following email:

From: Bartlett, Roscoe A Sent: Wednesday, June 08, 2016 12:31 PM To: 'trilinos-developers@trilinos.org' Cc: 'Trilinos Framework' Subject: Starting transition to the 'develop' branch

Hello Trilinos Developers,

I will now perform the transition to from the ‘master’ to the ‘develop’ branch. While this transition is occurring, you will not be able to push to the ‘master’ branch.

I will send out an email once the transition is complete (and you can push to ‘develop’).

Thanks,

-Ross

[Done]

b) Block who can directly push to ‘master’ on github repo (allow just a few people to push to 'master' initially).

I clicked "Restrict who can push" on the page:

https://github.com/trilinos/Trilinos/settings/branches/master

and added Brent, Jim, and myself to the allowed list. I then clicked "Save Changes".

[Done]

c) Update the github ‘develop’ to current tip of github ‘master’

I my local git clone where I had already created a local tracking 'develop' branch, I did:

[ (master)]$ git checkout develop
[(develop)]$ git pull
[(develop)]$ git merge github/master 
[(develop)]$ git push

Now we see:

[(develop)]$ git log -1 --name-status github/master 
commit abf1bd74ecdf74b10bc2b8a0f05fdfc1c9b08ec8
Author: Mark Hoemmen <mhoemme@sandia.gov>
Date:   Wed Jun 8 09:51:53 2016 -0600

    KokkosKernels: Fix unused typedef warnings in sparse mat-vec

M       packages/tpetra/kernels/src/impl/Kokkos_Sparse_impl_spmv.hpp

[(develop)]$ git log -1 --name-status github/develop 
commit abf1bd74ecdf74b10bc2b8a0f05fdfc1c9b08ec8
Author: Mark Hoemmen <mhoemme@sandia.gov>
Date:   Wed Jun 8 09:51:53 2016 -0600

    KokkosKernels: Fix unused typedef warnings in sparse mat-vec

M       packages/tpetra/kernels/src/impl/Kokkos_Sparse_impl_spmv.hpp

Looks good.

[Done]

d) Open up pushes to the 'develop' branch.

On the page:

https://github.com/trilinos/Trilinos/settings/branches/develop

I removed the check-mark beside "Restrict who can push to this branch" and clicked "Save Changes".

e) Send out email to trilinos-developers announcing that the transition is complete (again pointing to documentation and provide list of people they can ask for help).

I send out the following email:

From: Bartlett, Roscoe A Sent: Wednesday, June 08, 2016 12:45 PM To: trilinos-developers@trilinos.org Cc: Trilinos Framework Subject: Completed transition to the 'develop' branch

Hello Trilinos Developers,

The initial transition to the ‘develop’ branch is complete. To adjust, please follow the instructions at:

https://github.com/trilinos/Trilinos/wiki/VC-|-'develop'-'master'-workflow#transition_develop_master

If you have local changes on master, then please follow the instructions at:

https://github.com/trilinos/Trilinos/wiki/VC-|-'develop'-'master'-workflow#move_to_develop

If you have questions or run into problems, please send email to the trilinos-framework list for help.

Cheers,

-Ross

P.S. Now we need to transition to automated testing over to the new branch. That will happen next. This will result in some disruption in results going to CDash.

[Done]

f) Ask Brent to transition the nightly testing servers

I send the following email:

From: Bartlett, Roscoe A Sent: Wednesday, June 08, 2016 12:49 PM To: Perschbacher, Brent M Cc: Trilinos Framework Subject: FW: Completed transition to the 'develop' branch

Hello Brent,

Can you please make the transition of the Nightly testing servers over to test the ‘develop’ branch instead of the ‘master’ branch as we discussed yesterday (and documented in Trilinos GitHub #370) when it is convenient?

I will test a change to the CI server script to switch over to the ‘develop’ branch. Once I am sure that will work, I will get back to you when it is pushed and ready to use on the CI server machine.

Thanks,

-Ross

[Done]

bartlettroscoe commented 8 years ago

I updated and the script:

ctest_linux_continuous_mpi_opt_shared_sadl30906_jenkins.cmake

to pull from 'develop' and pushed the commit to the 'develop' branch. Now we just need to update the Jenkins driver to use this driver script and that should be it.

DETAILED NOTES:

Now I need to see about switching the CI server to pull from 'develop' instead of master.

From looking at the driver script:

Trilinos/cmake/ctest/drivers/sadl30906/ctest_linux_continuous_mpi_opt_shared_sadl30906_jenkins.cmake

which I found at:

http://testing.sandia.gov/cdash/viewNotes.php?buildid=2469381

I think that I should be able to just do:

diff --git a/cmake/ctest/drivers/sadl30906/ctest_linux_continuous_mpi_opt_shared_sadl30906_jenkins.cmake b/cmake/ctest/drivers/sadl30906/ctest_linux_continuou
index e3fe679..dcc2fc1 100644
--- a/cmake/ctest/drivers/sadl30906/ctest_linux_continuous_mpi_opt_shared_sadl30906_jenkins.cmake
+++ b/cmake/ctest/drivers/sadl30906/ctest_linux_continuous_mpi_opt_shared_sadl30906_jenkins.cmake
@@ -71,6 +71,8 @@ SET( CTEST_PARALLEL_LEVEL "10" )

 SET(Trilinos_ENABLE_SECONDARY_STABLE_CODE ON)

+SET(Trilinos_BRANCH develop)
+
 SET(EXTRA_EXCLUDE_PACKAGES Claps Optika TrilinosCouplings)

 SET( EXTRA_CONFIGURE_OPTIONS

Now to just test this locally, to see what the clone does:

$ env  CTEST_DASHBOARD_ROOT=$PWD \
  Trilinos_PACKAGES=Teuchos \
  CTEST_TEST_TYPE=Experimental\
  Trilinos_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE=Continuous \
  CTEST_DO_SUBMIT=OFF \
  CTEST_DO_UPDATES=TRUE \
  CTEST_START_WITH_EMPTY_BINARY_DIRECTORY=TRUE \
  CTEST_SUBMIT_CDASH_SUBPROJECTS_DEPS_FILE=FALSE \
  ctest -V -S \
    ../../../Trilinos/cmake/ctest/drivers/sadl30906/ctest_linux_continuous_mpi_opt_shared_sadl30906_jenkins.cmake

...

In the output I saw:

-- Trilinos_BRANCH='develop'

...

***
*** Update the source code repositories ...
***

Doing GIT update of '/home/8vt/localhome/PROJECTS/Trilinos.base/BUILDS/GCC-4.8.3/MOCK_CI_SERVER/Trilinos' ...
   Updating the repository: /home/8vt/localhome/PROJECTS/Trilinos.base/BUILDS/GCC-4.8.3/MOCK_CI_SERVER/Trilinos
   Use GIT repository type
   Old revision of repository is: abf1bd74ecdf74b10bc2b8a0f05fdfc1c9b08ec8
   New revision of repository is: abf1bd74ecdf74b10bc2b8a0f05fdfc1c9b08ec8
   Gathering version information (one . per revision):

CTEST_UPDATE(...) returned '0'

...

Doing switch to branch develop
Switched to a new branch 'develop'

...

Then looked and saw:

$ cd Trilinos/
$[(develop)]$ git status
On branch develop
Your branch is up-to-date with 'origin/develop'.
nothing to commit, working directory clean

Looks like gold.

I pushed the following commit to the 'develop' branch:

a6fe3ed "Pull from 'develop' instead of 'master' (Trilinos #370)"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Wed Jun 8 14:32:28 2016 -0400 (47 minutes ago)

M       cmake/ctest/drivers/sadl30906/ctest_linux_continuous_mpi_opt_shared_sadl30906_jenkins.cmake
bartlettroscoe commented 8 years ago

Since I don't own the Jenkins job I have handed this over to Jim and Brent to complete the transition of the CI server.

From: Bartlett, Roscoe A Sent: Wednesday, June 08, 2016 1:26 PM To: Perschbacher, Brent M; Willenbring, James M Cc: Trilinos Framework Subject: Update Trilinos CI server for 'develop'

Hello Brent and Jim,

I don’t know how you have the Jenkins server set up to drive the Trilinos CI build, but I updated the driver script to use the ‘develop’ branch instead of the ‘master’ branch. See:

https://github.com/trilinos/Trilinos/issues/370#issuecomment-224699484

Is there some place where the workings of the Jenkins driver for CI sever is documented or under version control? I don’t see anything about this under:

Trilinos/cmake/ctest/drivers/sadl30906/

Thanks,

-Ross

bmpersc commented 8 years ago

I have updated the nightly repo to default to develop and am in the process of cleaning up all the jenkins jobs.

jwillenbring commented 8 years ago

Forgive me if I missed the traffic, but has it been decided who will update the repo once per day manually the short-term, and what time of day that will happen?

bartlettroscoe commented 8 years ago

As per Mike's email below, I added:

https://github.com/trilinos/Trilinos/wiki/VC-|-'develop'-'master'-workfow-cheat-sheet

and added link on the wiki sidbar.

From: trilinos-framework-bounces@software.sandia.gov [mailto:trilinos-framework-bounces@software.sandia.gov] On Behalf Of Heroux, Michael A Sent: Wednesday, June 08, 2016 2:55 PM To: Trilinos Framework Subject: [Trilinos-Framework] Develop/master instructions

I want to mention that the following page is still too complicated:

https://github.com/trilinos/Trilinos/wiki/VC-%7C-'develop'-'master'-workflow#get_on_local_develop

For the 90+% of developers, I don’t see why they should have to scroll down the page to find the 1 or 2 commands needed to make the switch. It seems like we are blending background and theory with simple instructions, and the instructions are getting lost in the details.

Given that we have made it through the initial transition to develop (which is brain-dead simple and should not have required so much confusion), we should modify this page with the purpose of helping new developers learn about develop/master.

Mike

bmpersc commented 8 years ago

I have completed updating all the jenkins testing to wipe out their workspaces and ensure that they are running scripts from the develop branch. Those running their own testing that I know about have been contacted about they updates they need to make.

bartlettroscoe commented 8 years ago

I have completed updating all the jenkins testing to wipe out their workspaces and ensure that they are running scripts from the develop branch.

@bmpersc, did you also restart the CI server? It looks like you might have since a full CI build was started at 1:09 PM MDT shown here:

http://testing.sandia.gov/cdash/index.php?display=project&project=Trilinos&parentid=2471806

If so, the real confirmation will come when it starts the next CI iteration now that people have pushed to the 'develop' branch.

bmpersc commented 8 years ago

That run at 1:09 was before I had gotten to the CI build. However, before leaving last night I had done the necessary things to the CI build so that the first run of the day should be good. Looking at the workspace for the CI build this morning HEAD is set to develop so I believe everything should be good. Please let me know if it is not.

On a related note I don't believe that all of the builds done by developers have been updated yet so there may be a few stragglers. I was able to update the apollo and perseus tests though so a large portion of the developer tests have been updated.

bartlettroscoe commented 8 years ago

Looking at the workspace for the CI build this morning HEAD is set to develop so I believe everything should be good. Please let me know if it is not.

Looks like the CI server is pulling from 'develop' indeed. For more evidence, for example, in the first CI build today for Gtest in the configure output at:

http://testing.sandia.gov/cdash/viewConfigure.php?buildid=2472038

you see:

Trilinos repos versions:
--------------------------------------------------------------------------------
*** Base Git Repo: Trilinos
75ca038 [Wed Jun 8 19:29:12 2016 -0600] <rppawlo@sandia.gov>
Phalanx: add get_view and get_static_view to mdfield
*** Git Repo: preCopyrightTrilinos
c714979 [Thu Mar 31 14:12:16 2016 -0600] <amklinv@sandia.gov>
xSDKTrilinos: Added copyright statement to all source code
 --------------------------------------------------------------------------------

That is a pretty recent commit for Trilinos.

If you look at the current 'master', you see:

$ git fetch

$ git log-short --name-status -1 github/master 
abf1bd7 "KokkosKernels: Fix unused typedef warnings in sparse mat-vec"
Author: Mark Hoemmen <mhoemme@sandia.gov>
Date:   Wed Jun 8 09:51:53 2016 -0600 (23 hours ago)

M       packages/tpetra/kernels/src/impl/Kokkos_Sparse_impl_spmv.hpp

which is an older commit.

Now let's see what happens on the next CI iteration to make sure.

Next we need to figure out an interim update process for 'master'. I would like to propose we run the checkin-test.py script on the CI server sadl30906 using the exact same compilers, TPLs, etc. that are currently being used for the CI server. We will exclude some packages and some constantly failing tests (so the first update will work). That way, people can see the failures right on the CI build on CDash. We can use just a few processes so as not to overload the machine. The only question is who to send email to when it passes and when it fails? At the last meeting it was suggested that the trilinos-developers list should get these emails. We can set up a looping CI server that will be running constantly to make the updates very fast.

Anyone see a problem with this approach?

bmpersc commented 8 years ago

It looks like there is a problem with all the nightly testing, muir is still building from master. I believe I have fixed it. the Version.cmake file was explicitly telling all the testing to checkout the master branch. I have changed that to develop now so tonight's testing should be correct.

jwillenbring commented 8 years ago

@bartlettroscoe mentioned having projects maintain their own copy of Trilinos, pulling updates from master when tests pass to facilitate situations where someone wants to apply changes to Trilinos and client project simultaneously. I understand this concept at some level, but if people start committing their own changes to this repo, that complicates the workflow somewhat. Would the idea be to merge the next set of changes into the project's version of Trilinos, or would you snapshot over the top? What if there was a merge conflict in this automated process? It seems the process becomes significantly more complicated if it isn't just pulling from Trilinos a new version when tests pass, and not pulling if the tests don't all pass.

bartlettroscoe commented 8 years ago

I understand this concept at some level, but if people start committing their own changes to this repo, that complicates the workflow somewhat.

These workflows and issues are what I want to cover in the white-paper "Compare and contrast different technical approaches for multi-repository development and integration" for:

https://ideas-productivity.atlassian.net/browse/HOW-49

I need to take what have learned about these processes in the last 8+ years at SNL and then in CASL and write them down so others can benefit from them as well.

Otherwise, we can discuss specific use cases and come up with good workflows to address the challenges. With git and multiple repos and branches, there is away a way to do this well (if people will be willing to learn git and then give it a try).

jwillenbring commented 8 years ago

from @bartlettroscoe "Another issue that just occurred to me is what to do with the extra repos for inserted packages like CTrilinos, ForTrilinos, MOOCHO, Mesquite, Sundance, and extra repos like preCopyrightTrilinos which are listed in the file Trilinos/cmake/ExtraRepositoriesList.cmake. ... Any comments on this?"

I agree that this can wait until a later time. A ticket should be filed when this one is closed (or could be filed now really). Depending on what kinds of contributions we might get going forward, I am not certain that we want to impose a develop-master requirement on the extra repos. I see pros and cons. This topic warrants more discussion.

bartlettroscoe commented 8 years ago

So we can disable ROL tests for now. That will help the CI build.

From: Bartlett, Roscoe A Sent: Thursday, June 09, 2016 6:32 PM To: Ridzal, Denis Cc: Willenbring, James M; Perschbacher, Brent M; Kouri, Drew Philip Subject: RE: Clean up ROL CI failures?

Denis,

Okay, depending on how extensively these tests are failing, I may disable them for all configures or just for the CI build. But I will create a ROL GitHub Issue for this either way.

Cheers,

-Ross

From: Ridzal, Denis Sent: Thursday, June 09, 2016 2:04 PM To: Bartlett, Roscoe A Cc: Willenbring, James M; Perschbacher, Brent M; Kouri, Drew Philip Subject: Re: Clean up ROL CI failures?

Thanks Ross,

Drew and I are tied up at a workshop, and won't get to this until Monday.

Please go ahead and disable them (in any way you wish). Thanks again,

Denis

From: Bartlett, Roscoe A Sent: Thursday, June 9, 2016 13:53 To: Ridzal, Denis Cc: Willenbring, James M; Perschbacher, Brent M; Kouri, Drew Philip Subject: RE: Clean up ROL CI failures?

Denis,

You can disable them in ROL too if you would like. That will also result in not blocking people’s pushes if they do use the checkin-test.py script. If they are important tests you should eventually fix and re-enable them.

But if you don’t disable them in ROL, then we can disable them only for CI testing using:

-D <full_test_name>_DISABLE=ON

See:

https://trilinos.org/docs/files/TrilinosBuildReference.html#disabling-specific-tests

-Ross

From: Ridzal, Denis Sent: Thursday, June 09, 2016 1:49 PM To: Bartlett, Roscoe A Cc: Willenbring, James M; Perschbacher, Brent M; Kouri, Drew Philip Subject: Re: Clean up ROL CI failures?

Ross,

Would you prefer that we disable those tests in ROL?

How does the disabling just in the CI build work?

Thanks,

Denis

From: Bartlett, Roscoe A Sent: Thursday, June 9, 2016 9:06 To: Ridzal, Denis Cc: Willenbring, James M; Perschbacher, Brent M Subject: Clean up ROL CI failures?

Hello Denis,

We would like to use the current Trilinos CI build as the initial criteria for updating the ‘master’ branch. However, there are currently two failing ROL tests:

http://testing.sandia.gov/cdash/index.php?project=Trilinos&parentid=2472037

We can just disable them in the CI build and create a Trilinos GitHub issue if you would like for now.

-Ross

bartlettroscoe commented 8 years ago

I agree that this can wait until a later time. A ticket should be filed when this one is closed (or could be filed now really).

Done. See #440. We can discuss the details of this later and why you have to do it if you will be doing automated testing involving these extra repos in coordination with the Trilinos 'develop'/'master' workflow.

bartlettroscoe commented 8 years ago

I created Issue tickets for the failing STK (#441) and ROL (#442) CI tests. The next step will be to disable these tests and verify that the full CI build of Trilinos on this machine sadl30906.srn.sandia.gov passes. Then I can set up a checkin-test.py driver script to run this build.

bartlettroscoe commented 8 years ago

After going down over the weekend, the Trilinos CI Build seems to be processing the commits being pushed to the 'develop' branch just find. For example, see this query for the CI results from yesterday.

@bmpersc, have the issues with Nightly testing of the 'develop' branch been resolved yet? It looks like since you made the following commit, that CTest drivers should be using the 'develop' branch (reguardless if they are pulling from the 'nightly' Trilinos repo on SSG or not).

commit f70fcbd54a42764f8a0571b72ba20580d8771cd1
Author: Brent Perschbacher <bmpersc@sandia.gov>
Date:   Thu Jun 9 08:49:54 2016 -0600

    Update nightly testing branch to develop

diff --git a/Version.cmake b/Version.cmake
index 6c83aef..fbb08ba 100644
--- a/Version.cmake
+++ b/Version.cmake
@@ -66,5 +66,5 @@ SET(Trilinos_VERSION_STRING "12.7 (Dev)")
 SET(Trilinos_ENABLE_DEVELOPMENT_MODE_DEFAULT ON) # Change to 'OFF' for a release

 # Used by testing scripts and should not be used elsewhere
-SET(Trilinos_REPOSITORY_BRANCH "master" CACHE INTERNAL "")
+SET(Trilinos_REPOSITORY_BRANCH "develop" CACHE INTERNAL "")
 SET(Trilinos_TESTING_TRACK "" CACHE INTERNAL "")

Therefore, I am going to assume that all of the automated testing is switched over to the 'develop' branch.

bmpersc commented 8 years ago

Yes as far as I can tell the nightly testing is all working from the develop branch now. I did have to wipeout the work space again for the Trilinos_REPOSITORY_BRANCH to take affect, but I did that for all the testing I have access to.

bartlettroscoe commented 8 years ago

I did have to wipeout the work space again for the Trilinos_REPOSITORY_BRANCH to take affect, but I did that for all the testing I have access to.

That is strange that you had to delete the repo to have this take effect. Looking at TribitsCTestDriverCore.cmake, it would appears that setting Trilinos_REPOSITORY_BRANCH=develop should do the trick (unless some other script is setting Trilinos_BRANCH in the env or something).

bartlettroscoe commented 8 years ago

A brief update on the status of this story:

So what is it going to be? Should we switch to --no-ff merges or not?

But before we make the switch to use --no-ff merges, it would be good to write a short little shell script to create the amended merge commit on the 'master' branch that you would call like:

$ cd Trilinos/
$ git checkout master
[(master)]$ git pull   # from origin/master
[(master)]$ ./commonTools/git/merge-develop-to-master.sh  <good-sha1>
[(master)]$ git push   # to origin/master

You would just feed in the known good commit <good-sha1>. I can write a quick Shell or Python script for this if there are no objections.

bartlettroscoe commented 8 years ago

The issue #441 and #442 are no longer blocking as these failing STK and ROL tests have been disabled or otherwise been removed from the set of BASIC pre-push CI tests. The tests that have been disabled will not be re-enabled until they are fixed.