search

huserben / TfsExtensions

Extensions for TFS 2015+ such as custom Widgets (require TFS 2017) and Build Tasks
MIT License
45 stars 22 forks source link

Wait for build task to finish prevents UI refresh #23

Closed JohnHennesey closed 6 years ago

JohnHennesey commented 7 years ago

First off - love this extension. V1 was great, V2 is even better!

The addition of the 'pause until task completes' solved an issue for us, but while it goes into the 'sleeping for xx seconds' loop it does something to the TFS website where it doesn't allow any other browser tabs to refresh. This means I am unable to do anything else in the web UI until my 2-3 hour master build finishes. Not a deal breaker by any stretch as the master build normally kicks off at 1 am, but in the event there is a failure and I need to build a package in the morning it would be nice to be able to get the attention of the website. sleeping loading

Update: If there is a task mixed in the middle that does not have this option selected the UI will handle all waiting requests. But further down the line I have more builds that wait, the UI goes back into the wait state. It's definitely related to the sleep.

JohnHennesey commented 7 years ago

I think I isolated our build issues down to this - after several reboots of the controller and build machine the patter emerged. With the latest version of this extension installed it seems to hose up communication with our TFS controller (2017.03). When I disable the extension - same thing. I had to uninstall the extension and all our builds work again. I then re-installed it, and it went back to the same invalid state.

Oddly enough it is triggering new builds, it just seems to distract the webUI enough that it cannot respond to any other requests.

How do I get back to v1 of this? Many people are mad at me right now :)

JohnHennesey commented 7 years ago

Update: I was able to uninstall and revert back to 1.8.5, the fire has been extinguished.

huserben commented 7 years ago

Hi John I'm sorry to hear that the extension is causing problems. At least you managed to revert to the old one.

So just to get it right - as soon as you install the extension you start to have problems, even without using the extension or just when it is actively waiting for a build to finish?

I can try to reproduce this later this week(end).

You think it is the sleep itself and not the amount of requests that are the problem right? Looking at your screenshot it will always wait for 60 seconds to send one request - if this is a problem for the TFS controller we are in trouble :D

I do not have much experience with TypeScript/Node.js but somehow it feels hard to image that this single sleep will cause so much problems. I really just used the answer provided here in this SO Post: https://stackoverflow.com/questions/37764665/typescript-sleep

huserben commented 7 years ago

Hi @JohnHennesey at least with my VSTS environment I could not reproduce the problem after letting a build wait for 30min for another one to finish everything behaved as usual...

JohnHennesey commented 7 years ago

Thanks for the quick response - I am in good shape now (still on v 1.8.5.). Here is a replay of what happened:

  1. Try to install v2.1 - TFS prevented saying it already existed
  2. Uninstall v1.x
  3. Install 2.1
  4. Change over all build tasks to include new 'wait for xx to finish' functionality
  5. Overnight build kicked off
  6. Master build starts, starts calling child builds
  7. Master build times out after 60 minutes. This wasn't a problem before because it could submit all child builds (15) in about 20 minutes. Now it runs serially, so it timed out.
  8. Still appeared child builds were kicked off even though it timed out
  9. Child build failed
  10. Fix bad code in child build, Update timeout to 5 hours
  11. Kick off main build again
  12. Made it 90% of the way through and the TFS website stopped responding to webapi requests
  13. Reboot both build & controller machine
  14. Kick off new builds
  15. Webapi stops responding
  16. Repeat reboot & retry about 5 times
  17. Disable kick off build extension task
  18. Api still not responding
  19. Reboot
  20. Kick off new builds
  21. Api stops responding
  22. Reboot
  23. Uninstall extension
  24. Kick off builds - all is happy.
  25. Install extension
  26. Api stops responding (but I didn't kick off a build!)
  27. Figure out I can restart the app pool in IIS vs. reboot (love it! - better than TFS 2013)
  28. Restart app pool, try web site, api stops responding (repeat 4 or 5 times)
  29. Uninstall extension
  30. Install 1.8.5, edit master build task so it reflects json parameters for 1.85, Save
  31. Kick off master build -- everything works perfectly.

I looked through event viewer and found only entries about 404 when trying to use the WebApi's - I can get the exact ones for you if you wish. Once again I am good now - and I still love this extension. :)

If I can answer any more questions, or try reinstalling it again please let me know. Now that I know how to back it out and install an older version I am feeling more comfortable with experimenting.

John

huserben commented 7 years ago

Hi John thanks for the extensive description of what you did. It still seems strange that it stops working as soon as the extension is installed even though no build was kicked off... If you say the WebAPI stops responding, do you mean just for the FrontEnd or does the build as well fail when waiting for the other build to finish? As the Task uses the regular TFS WebAPI I would imagine that this should fail as well...

If you have access it would be interesting to see the event viewer when it first stopped responding.

I currently don't have any Test Environment where I could try to reproduce that problem so it's hard for me to judge what is really going on.

In the 1.* version there is also the option to wait for the build to complete, did you enable this in your current setup?

I really want to get to the bottom of this because staying on the 1.* version is probably not the best solution as I don't plan to maintain this any longer...

huserben commented 7 years ago

So another update - I just installed TFS 2017 Express locally on my private Desktop, installed the Task and let it a build run and wait for over 1h (checking every 10 seconds) I could not see any probelm in the Web UI of the TFS and I don't have any errors in my event log :-/

JohnHennesey commented 7 years ago

Thank you so much for looking into this. Once our sprint ends I plan to reconfigure the build agents (or just reinstall them) and do a repair on the TFS machine. I have seen some instability over the last week that I think is a bit sketchy.

I will keep you posted - hope you had a great weekend. John

huserben commented 7 years ago

Hi John

thank you for reporting such things instead of silently revert back to the old version :-)

Currently I don't have too much time to look deeper into the issue myself as I'm doing it after my regular work. But please let me know if you find anything interesting that could be of help. If there is something I can do to assist you I'm more than happy to help.

JohnHennesey commented 7 years ago

This may be related to it - I think I found a bug in how our builds are assembled. Once our sprint ends I plan to upgrade to 2.x and will let you know. With dynamic repository paths I saw all kinds of weirdness in the UI experience.

https://developercommunity.visualstudio.com/content/problem/101267/dynamic-repository-path-breaks-ci.html

huserben commented 7 years ago

Ok, that sounds interesting, I wasn't even aware that you can use dynamic repository paths. Please let me know when you found out something :-)

huserben commented 6 years ago

Hi @JohnHennesey

so were you able to figure out more about this weird behavior?

JohnHennesey commented 6 years ago

Unfortunately I have not. I'll keep you posted.

huserben commented 6 years ago

Hi @JohnHennesey

I just had an idea what might could help as well in this problem - at least to reduce a bit the load towards the TFS REST API. However it would mean splitting your build definition into two:

So it should be possible now to trigger a number of builds and store the ids of the Triggered Builds id. Then after you triggered all your builds you just trigger a new build of your new Build definition, and as a parameter you will pass the triggered build ids. In the other build definition you will use the "WaitForBuildsToFinishTask" and await the builds as usual and after this is done you just execute whatever was in the first build definition after all the builds were done. (For this just add as a build variable "TriggeredBuildIds" and make it settable at queue time so we can parametrize it from the other build)

This would have the advantage that you as well "free" the agent that runs the initial build because we don't need poll all the time. Instead it will just be put in the Build Queue and once it's up the chances that the other builds already finished or are close to finish should be quite high (depending on the number of build agent you use that is).

Anyway I'm not sure if you are still looking into the issue or have time at all, it just popped into my mind today that this "splitting" of build definitions is now possible so we can trigger in one but await in another :-)

JohnHennesey commented 6 years ago

Thanks for the suggestion! We found a solution that covers us for now. Should I need to revisit it I will definitely keep it in mind - creative!

huserben commented 6 years ago

Hi John

ok, good to hear. So do you consider this issue to be solved then or should we still keep it open?

JohnHennesey commented 6 years ago

I am good - go ahead & close it.

Check this out... MS is dabbling in this functionality now.

https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/2165043-provide-build-configuration-dependencies-in-tfs-bu?tracking_code=8cb7cd5f7f7370cb934cfbd418407688

huserben commented 6 years ago

Alright, will do. Just create another issue if you stumple upon something again :-)

Yeah I saw, let's see how they plan to solve it :-)