Why after connecting to VSO is everything taking so long?

[guardrex]

Is there still some on-going work with VS 2015 and VSO processing? Ever since adding some projects to VSO, VS is super slow and hanging occasionally with simple tasks, such as copying a file, moving a file, and deleting a file. It either takes anywhere from 5-20 seconds or actually (occasionally) just hangs. This only occurs with projects I have in VSO.

[sayedihashimi]

@GuardRex is this specific to ASP.NET 5/DNX projects or for all project types?

[guardrex]

Not sure on that ... I'm only working Core CLR projects actively with VSO at the moment. I'll be back at the office this afternoon and will test with an old .NET 4.x project and report back. On Nov 9, 2015 10:34 AM, "Sayed Ibrahim Hashimi" notifications@github.com wrote:

@GuardRex https://github.com/GuardRex is this specific to ASP.NET 5/DNX projects or for all project types?

— Reply to this email directly or view it on GitHub https://github.com/aspnet/Tooling/issues/271#issuecomment-155148881.

[sayedihashimi]

@GuardRex ok if you could do that it would be great. Thanks.

[guardrex]

@sayedihashimi

It's generally quick with an old .NET 4.5.1 project I have in VSO. I didn't notice much latency on copying, moving, and deleting a file.

I went back into the Core CLR project, and I added a .tfignore file with a \node_modules entry in the file. There are >3,600 files in the node_modules folder. It seems snappier now.

Does VS need to traverse all the files and folders of a project when it's dealing with VSO? Does it skip folders if they are indicated in .tfignore? If so, this might explain why its running faster now.

[guardrex]

@sayedihashimi

Still happening (at least occasionally). It's intermittant.

In this example, I'm just adding a script from the HD to the scripts folder of the project. After a few minutes waiting, it failed with this exception. BUT ... It did actually add the script to the folder when I dismissed the exception pop-up.

This is with the Core CLR proj in VSO. Is there anything I can do to examine why these types of delays are happening? I still have the gut feeling this is related to VSO somehow.

[guardrex]

@sayedihashimi Another hang ... all I was doing was trying to delete three font files out of wwwroot. Spinning cursor ... perhaps 3-5 minutes go by ... then ...

If there is any data I can capture for you, let me know. It's still is only happening rarely, but it seems to be an issue with my VSO-based projects. This proj does have a .tfignore with a \node_modules in it.

[guardrex]

Oh ... and it DID delete the three font files in the process. The operations seem to work, but the delay is long and that exception is the common outcome when this happens.

[sayedihashimi]

@guardrex when you run into a hang if you can get a dump file that would be ideal. That would speed up the investigation.

[guardrex]

@sayedihashimi ... Using this kit? Download kits and tools for Windows 10

https://msdn.microsoft.com/en-us/windows/hardware/dn913721.aspx

[sayedihashimi]

You just need VS see https://msdn.microsoft.com/en-us/library/d5zhxt22.aspx for more info.

[guardrex]

@sayedihashimi Thanks ... I'll get to it asap ... probably this weekend and report back.

[sayedihashimi]

@GuardRex ok great thanks.

[guardrex]

I just noticed the messages appearing in the Output window after opening a solution ...

I'm going to try to get to that dump file today.

[sayedihashimi]

Thanks, I haven't seen this before.

[guardrex]

@sayedihashimi [EDIT] Took four of them and noted in the filenames the action that I was performing.

How do I get these to you securely?

[guardrex]

ping @sayedihashimi

I saved a fifth dump file ... spinning cursor for several minutes merely editing a view file. It might be having trouble connecting to VSO, because all of the files show red checks next to them in the UI; however, it really shouldn't hang for several minutes if there is a connectivity issue with VSO IMO.

What's the best way to get these dump files securely to you?

[sayedihashimi]

You can upload to OneDrive/Dropbox and then mail me a link sayedha@microsoft.com

[abpiskunov]

I copied dumps here: \vwdbuild01\Temp\antonpis\dump\VSO @GuardRex what VS and WTE versions do you have? Also do you have U1 installed?

[guardrex]

@abpiskunov If the VS data below doesn't have it, let me know where to check for the version numbers you need. AFAIK, I do have U1 installed, but where can I check that?

Microsoft Visual Studio Community 2015
Version 14.0.23107.0 D14REL
Microsoft .NET Framework
Version 4.6.00079
Installed Version: Community
Visual Basic 2015   00322-20000-00000-AA062
Microsoft Visual Basic 2015
Visual C# 2015   00322-20000-00000-AA062
Microsoft Visual C# 2015
Visual C++ 2015   00322-20000-00000-AA062
Microsoft Visual C++ 2015
Application Insights Tools for Visual Studio Package   1.0
Application Insights Tools for Visual Studio
ASP.NET and Web Tools 2015 (Beta8)   14.0.61012.0
ASP.NET and Web Tools 2015 (Beta8)
ASP.NET Web Frameworks and Tools 2013   5.2.30624.0
For additional information, visit http://www.asp.net/
Common Azure Tools   1.6
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.
GenerateUnitTest   1.0
Generates unit test code for methods in classes under test.
GitHub.VisualStudio   1.0
A Visual Studio Extension that brings the GitHub Flow into Visual Studio.
Indent Guides   14
Indent Guides
Adds visual guides at each indentation level.
Microsoft Azure HDInsight HQL Service   2.0.2600.0
Language service for Hive query
Microsoft Azure HDInsight Tools for Visual Studio   2.0.2600.0
An integrated development environment for HDInsight application development.
Microsoft Azure Mobile Services Tools   1.4
Microsoft Azure Mobile Services Tools
Microsoft Azure Tools   2.7
Microsoft Azure Tools for Microsoft Visual Studio 2015 - v2.7.30818.1601
NuGet Package Manager   3.2.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.
PreEmptive Analytics Visualizer   1.2
Microsoft Visual Studio extension to visualize aggregated summaries from the PreEmptive Analytics product.
Python Tools for Visual Studio   2.2.30718.00
Python Tools for Visual Studio provides IntelliSense, projects, templates, Interactive windows, and other support for Python developers.
Python Tools for Visual Studio - Django Integration   2.2.30718.00
Provides templates and integration for the Django web framework.
Python Tools for Visual Studio - Profiling Support   2.2.30718.00
Profiling support for Python projects.
SQL Server Data Tools   14.0.50730.0
Microsoft SQL Server Data Tools
Web Essentials 2015   0.5.197

[sayedihashimi]

@abpiskunov U1 is not public so he doesn't have it installed.

[abpiskunov]

TFS #166982

[guardrex]

@abpiskunov I have two more samples (dumps) if needed: One is simply editing a JavaScript file and it froze (it didn't recover from this one). The other is a very long check-out operation from VSO. It ended after a few minutes with the same exception posted above "The local data store is in use by another operation," but it was able to check out the file.

I don't think you'll need these, but I'll hold on to them just in case you need more samples.

[guardrex]

@abpiskunov I don't know if a hotfix of some sort can be made available when you've got this one licked, but this one is killing me a slow death. VS is freezing regularly (several times an hour) as I work. These 3-5 minute freezes are killing my productivity. I can take the proj out of VSO, but that's not particularly safe vis-a-vis source control backup of my projects.

[abpiskunov]

Unfortunatelly it looks like the problem might not be even in our (WTE) code, but in other components. I looked at dumps and will reroute the bug I opened to other teams to see what they could find. If it (unlikely) turns out our bug and returns to us we might produce some privates (need to check if it is legal or not at all). However if it is in other components , they are tied to VS schedule update 2 or Dev15.

[guardrex]

@abpiskunov Looks like my best bet is to ditch VSO for right now (if that's the problem ... this seemed to start happening when I put these projects into VSO and doesn't seem to happen with some projects that are not in VSO) ... just perhaps copy compressed projects to an offsite server for backup purposes at the end of each day.

Thanks for looking into it so quickly and getting back to me.

[guardrex]

@abpiskunov I want to mention that things have been better since going to VS Update 1. It's still a hair sluggish, but I think that's just due to our lousy Internet connection at this office (TelePacific = garbage ISP service).

The exception for "local data store in use" seems to happening much less (if at all now). Were the dumps I sent still being looked at? If they're not being analyzed right now, perhaps I should close this ... I can re-open later if it starts hanging again with that exception.

[abpiskunov]

I am not sure, bugs are assigned to other teams and they will take it from there. Let us know if you see more problems.

[guardrex]

@abpiskunov Since its on TFS #166982, is it best to leave this Github issue open or close now? That exception ... TF400030 ... seems to have gone away for me since VS Update 1. I'll re-open if I see it cropping up again.

[guardrex]

@abpiskunov Spoke too soon. I just tried to add a class file to a project, and it did it again ... hung. Besides, I guess since its on your internal TFS that this should stay open while that's being looked at.

[guardrex]

@abpiskunov Arrrrrgh ...

Any news yet on TFS #166982 ... I'm still having to murder VS with Task Manager several times a day to get back to work.

[abpiskunov]

No, bug is under investigation of the other team. In any case fix will not be public for a while, most likely until VS15 U2 is released...

[guardrex]

Ok, thx for the update.If it gets too crazy bad, I'll drop VSO for a while, which seems likely. It drives you nuts when VS freezes up.

[abpiskunov]

@GuardRex is it possible to share dumps again? CPS team investigating it now.

[guardrex]

@abpiskunov Yes, I think ... I think I have them here. I wish I would have named them a little more clearly than devenv1.DMP and devenv2.DMP, but I think these are the dumps (file mod dates are 11/24 on these, which matches above).

What's the best way to get these to you? Post and E-mail the link to @sayedihashimi ?

[guardrex]

They aren't attached to your internal Bug #166982 ?

[lifengl]

@GuardRex can you still reproduce this issue? I would like to do some investigation for this problem. If you still can reproduce it, can you take an ETW trace, and share with us. You can use PerfView tool (http://www.microsoft.com/en-us/download/details.aspx?id=28567), which is a great tool to investigate performance problems. You need turn on the merge/zip option when you take the trace, so it can be zipped into one file and be usable on another computer. You can start to take the trace when VS is frozen, and take 30 seconds, and stop it. Don't need trace the whole time, because the file can grow too big. It will tell us what VS is doing in that period of time. Unlike the dump, which may catch or may miss the problem.

Let me know whether it works for you.

[guardrex]

@lifengl I completely uninstalled VS and went to VS Code b/c VS+VSTS was giving me too many problems. Therefore, I can't reproduce. I guess whatever I sent (dumps) that should be attached to your Bug #166982 isn't what you need. Should we close and hope that new tooling for RC2 eliminates the issue? I don't see too many others complaining about VS+VSTS. Maybe it was just my setup.

[lifengl]

@GuardRex Do you still have those dump files? If you still have them, can you share it with us again. Somehow, the original links to access those dump files do not work any longer.

BTW, based on the pervious conversation, you are using TFS through VSOnline, right? and this is a random problem happening when you do some project modifications, including moving a single file, right?

[lifengl]

BTW, can you tell us the size of your project, and the size of the folder? when you had this problem, have you tried a smaller project to see whether the problem happened or not?

[guardrex]

@lifengl Hi ... sorry for the delay ... just got into the office.

On the dump files, I deleted the originals because I thought they had pulled them down and attached them to the internal bug. I don't have those, but I do have two more recent dumps. These are small enough that I should be able to E-mail them directly to @sayedihashimi referencing this issue. If I try to put them online again, we have horrible TelePacific Internet here, and it will take an hour to upload them.

Yes, it was VSTS, and I was using VS 2015 Update 1 with VSTS directly. Our on-prem TFS is not part of the version control for these projects ... I was only using the VSTS for these. It was usually hanging on operations with files and folders creating, deleting, renaming, and moving. Merely opening, closing, and saving didn't seem to cause a problem.

These projects are small. I think they're generally <10MB in size (that's excluding bin assets and just considering the project files; the bin was never subject to source control AFAIK and doesn't appear in VSTS in the online GUI).

I'll see if I can E-mail those dumps now.

[guardrex]

@lifengl Oh ... I forgot how large these dump files are. I'll upload to storage and E-mail the links and ping back here when they're there.

[guardrex]

@lifengl I just sent the link to the dump files to @sayedihashimi by E-mail.

[lifengl]

@GuardRex how many projects do you have in the same TFS workspace, and how many files do you have. A TFS local workspace doesn't work very well, when it has a large amount of files.

Unfortunately, those dump files are 64bits, don't work very well for 32bits processes, and limited what we can extract from them.

[guardrex]

@lifengl I have 24 team projects in the default collection. I don't see where I can get a file count in the VSTS UI. Regarding the local workspace, I'll make sure when I get back into VS at RC2 time that I exclude the bin and node_modules.

I'm going to close now, and I'll re-open if these issues reappear with VS (RC2) and VSTS. Thanks for your help.

[lifengl]

@GuardRex thank you a lot to provide lots of information for us. Based on the dump, it looks like it is a lock contention (to access TFS workspace) between the UI thread and a background to scan the workspace after some pending changes. The TFS local workspace can have performance problem, when it has a large amount of files. We guess that might be the source of this problem.

Thanks