search

csoltenborn / GoogleTestAdapter

Visual studio extension that adds support for the C++ testing framework Google Test.
Other
143 stars 100 forks source link

Remove bogus "Working directory" option from settings #308

Closed pps83 closed 4 years ago

pps83 commented 4 years ago

This is a copy from https://github.com/microsoft/TestAdapterForGoogleTest/issues/172 I tried this original GTA and it has exactly the same problem.

This sample project shows this problem: https://github.com/pps83/BrokenGoogleTestAdapter In my case, my code uses relative path during tests, more over, there are globals that use that relative path. Because GoogleTestAdapter doesn't use project's working directory path, but instead uses some random working dir, code that uses relative path suddently doesn't work in Test Explorer (yet, if I run the test from VS then everything is green). In the project above, you can try both project, they both pass. However, from GTA tests do not work, and one of the projects shows that discovery doesn't work. Also, when I tried to set $(ProjectDir) which is the default in VS for a project's working dir, then test explorer doesn't even work

pps83 commented 4 years ago

As a side note, I'm intermittently getting "No source available" for tests. If I use release builds of my unit test project then I see source location, in debug builds I have "No source available". I use vs2019

csoltenborn commented 4 years ago

Wow, a "BrokenGoogleTestadapter" GitHub project :-) - thanks for your efforts of trying to make this easily reproducable for me!

However, I just had a quick look, but cannot reproduce the behavior you are seeing:

I do not see any problem with GTA right now (and, btw, obviously GTA does not use "a random working dir" ;-) ).

image

pps83 commented 4 years ago

However, I just had a quick look, but cannot reproduce the behavior you are seeing:

I accidentally fixed the problems (on autopilot) when committed the code :) Please pull updated version and you'll see it.

I do not see any problem with GTA right now (and, btw, obviously GTA does not use "a random working dir" ;-) ).

When I say random, I mean that it's not what a project specifies in project's Debugging section. By default, it's always $(ProjectDir) in visual studio. With GTA, I cannot even set that directory in settings (using this env variable). If I clear that working dir in GTA settings, then C:\windows\system32 is a working dir (why!?)

csoltenborn commented 4 years ago

That's true and not true at the same time :-)

First of all, you are right in that GTA does not make use of a project's settings by default. It has always been like that, it's clearly pointed out in the docs, and the reason simply is that I wasn't able to figure out how to access these settings from within GTA - Visual Studio's documentation is pretty bad with respect to its internal APIs, and I only have a limited amount of time ;-)

This is certainly not nice, and in fact I recently have seen some instructive code that might help me to improve on that (however, I do not have much time for GTA at the moment - feel free to provide a PR!). But there's a workaround since a couple of months - see docs and SampleTests solution.

Finally, the default setting for the working directory is the executable's directory. Apparently, this has been enough for almost all users. And, by the way, it should be pretty easy in most cases to get your project's directory from there (e.g., try$(ExecutableDir)..\.. as the working dir).

If you clear the working dir setting, then C#'s Process class is invoked with an empty working dir if I remember correctly - I guess that's the reason why you end up with C:\Windows\system32 on your machine (but I'm not sure about that).

pps83 commented 4 years ago

First of all, you are right in that GTA does not make use of a project's settings by default. It has always been like that, it's clearly pointed out in the docs, and the reason simply is that I wasn't able to figure out how to access these settings from within GTA - Visual Studio's documentation is pretty bad with respect to its internal APIs, and I only have a limited amount of time ;-)

I thought that was the reason. GTA is now forked by microsoft, I guess they have all the resources to figure it out

Finally, the default setting for the working directory is the executable's directory. Apparently, this has been enough for almost all users. And, by the way, it should be pretty easy in most cases to get your project's directory from there (e.g., try$(ExecutableDir)..\.. as the working dir).

If that was per-project config, that would be OK, and, certainly, it would be better to add $(ProjectDir) as well.

csoltenborn commented 4 years ago

MS has mainly invested in localization and security - additionally, GTA can be used from VS 2013 on, whereas TAfGT only supports VS2017 and newer, so it's not clear whether I could make use of a solution implemented by them.

I don't know what you mean by "per-project config" (S(ExecutableDir) obviously is project-specific, and you can provide different settings for different projects). If it would be possible to povide a $(ProjectDir) placeholder without evaluating the project-specific VS settings, i would have done that - unfortunately, to my knowledge it is not.

Can I close this issue? I haven't looked at your example solution again, since I guess I have provided all information needed to fix your problems... Let me know if that's not true ;-)

pps83 commented 4 years ago

MS has mainly invested in localization and security - additionally, GTA can be used from VS 2013 on, whereas TAfGT only supports VS2017 and newer, so it's not clear whether I could make use of a solution implemented by them.

I don't know what you mean by "per-project config" (S(ExecutableDir) obviously is project-specific, and you can provide different settings for different projects). If it would be possible to povide a $(ProjectDir) placeholder without evaluating the project-specific VS settings, i would have done that - unfortunately, to my knowledge it is not.

I just tried AllTestSettings.gta.runsettings and it doesn't work. More over, it's fairly broken and I had hard time to get it to work. In my case project work dir happens to be ${SolutionDir}\vs and I used this config to set it:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <GoogleTestAdapterSettings>
    <SolutionSettings>
      <Settings>
        <WorkingDir>${SolutionDir}\vs</WorkingDir>
      </Settings>
    </SolutionSettings>
    <ProjectSettings/>
  </GoogleTestAdapterSettings>
</RunSettings>

It's fairly broken because it didn't work if I tried to set ${SolutionDir}/vs and I had no idea why it didn't work. But after trial and error I figured it out and ${SolutionDir}\vs worked, Except, it doesn't work at all. It does use this dir to run unit test from, however, it seems that before running it GTA sets that bogus working dir from global settings. In short, if WorkingDir is not a valid dir, then it cannto run anything and doesn't give a good log why it doesn't work. Once GTA is able to cd to WorkingDir it will run unit test from this dir, but with some wrong cwd. If I _getcwd from my unit-test I get directory where the test exe is located

pps83 commented 4 years ago

Seems that the AllTestSettings.gta.runsettings example is wrong and there is no such thing as ${SolutionDir}. If I set it to $(ExecutableDir)\..\.. (note, there must be a slash after $(ExecutableDir)) then it work.

Update: $(SolutionDir) works, but not ${SolutionDir}

csoltenborn commented 4 years ago

Oops, sorry for that - it's always $(Placeholder) (see docs))... Note that $(SolutionDir) is not available out of the box if GTA is installed via Nuget (e.g. in case you have CI).

What exactly is wrong with AllTestSettings.gta.runsettings? I don't see the problem...

csoltenborn commented 4 years ago

I have just now seen your other comment, and this kind of starts to become slightly annoying, I must admit :-| Please note that there are thousands of users who use GTA without any problems, so maybe the fact that you a have hard time getting it to run is not only caused by GTA.

Please study the docs as well as the SampleTests solution (which makes use of all kinds of configuration possibilities and works just fine) before complaining any more about about "things being fairly broken", "doesn't work at all", "bogus working dir", "poor logging" (for the latter, I'm sure you have discovered the different log levels). Or feel free to use any other, better working test adapter.

I'm closing this issue now. Feel free to open new issues for any concrete questions you still have after having read the complete documentation and having looked at the SampleTests solution.

pps83 commented 4 years ago

I have just now seen your other comment, and this kind of starts to become slightly annoying, I

I'd lie if I said that it's slightly annoying when docs/examples are clearly misleading. As I said, AllTestSettings.gta.runsettings example uses ${SolutionDir} which will never work.

I'm closing this issue now. Feel free to open new issues for any concrete questions you still have after having read the complete documentation and having looked at the SampleTests solution.

Questions that I have: 1) In my solution I have tens of different executables. Only one of them is unit test. I'm not sure how exactly unit test detection works, but I absolutely don't want some of these exes to run in the background by the test detector. It seems that detector somehow knows that these aren't unit tests and doesn't run them. Is there a way through config to specify unit test executable/project, so that other projects wouldn't be touched (and wouldn't be build) when I request test discovery from test explorer.

2) I'm looking how to configure working dir, so that a freshly cloned repo would work output of the box. On top of that, I'd really like to have it working with GTA and TAFGT. .runsettings doesn't work, as it requires manual selection from Test menu. This isn't acceptable for me. Seems like that only way that works is the SolutionName.gta.runsettings placed along with solution file. Is there another way that works with both of them? I kind of like settings-helper-files but it works only with GTA.

csoltenborn commented 4 years ago

Just to get that straight: From that single wrong occurrence of the placeholder (with 2 correct usages in the same file and dozens of other correct usages in the repository), and ignoring the rest of the documentation (in particular of the Working directory option) and the SampleTests solution, you get confused and derive that the documentation is "clearly misleading"? That tells more about you then about the documentation's quality, I guess :-)

To your questions:

  1. RTFM
  2. No