search

jediwhale / fitsharp

Functional testing tools for .NET
http://fitsharp.github.io
Other
152 stars 73 forks source link

How to map all the code referenced from the Fitnesse tests? #177

Closed MarkKharitonov closed 1 year ago

MarkKharitonov commented 2 years ago

Rationale: We want to refactor our Fitnesse code (we use .NET and fitSharp). However, some methods are called from the Fitnesse through reflection and these methods make it very hard to do refactoring safely.

Is there a way to map all the code invoked from Fitnesse? Ideally, we would have some kind of an object model we can inspect or a visitor we can use and get called back for every property or method mentioned in the content.txt files.

Is it possible?

jediwhale commented 2 years ago

It may be possible. I'll take a look once I finish the other issue I'm working on.

MarkKharitonov commented 2 years ago

Much appreciated. Looking forward. BTW, if the issue you working on is calling extension methods, then I am looking forward for that too :-). Again, thank you very much.

jediwhale commented 2 years ago

See release 2.8.4 https://fitsharp.github.io/FitSharp/Release284.html

MarkKharitonov commented 2 years ago

Hi, This is good, but I was hoping we could have a parser logic exposed. You see running all the Acceptance Tests locally is not feasible. So this option will have to be enabled in the QA environment. Then the logs to be collected and aggregated. All doable, but painful.

If we had an ability to parse the forest of the content.txt files and just map the methods invoked by them, that would be so useful.

Now maybe this is impossible in general, because method names could be extracted from variables with values known only at runtime - much like reflection vs static code. That is OK. Mapping static code is still very useful. The parser could report for every method call whether it is known at the parse time and if affirmative - output its name and arity. Even if multiple overrides with the same arity exist, it is still extremely useful.

jediwhale commented 2 years ago

Finding method names from the parser output is not possible. The parser just outputs a tree of strings. It's only at runtime that each fixture or keyword selects which strings are class names, method names or parts of method names, or data. In other words, the fitSharp environment is 100% dynamic execution, 0% static.

When I get some more free time, I can provide additional information in the log, like number and types of parameters.

MarkKharitonov commented 2 years ago

Maybe our case is different from the general one, I do not know. For us, most of the method and fixture names in content.txt are done using constant strings, so it is possible to know which methods are called on which fixture. The example from http://fitnesse.org/FitNesse.FullReferenceGuide.UserGuide.WritingAcceptanceTests.SliM.ScriptTable demonstrates it very clearly.


script | login dialog driver | Bob | xyzzy -- | -- | -- | -- login with username | Bob | and password | xyzzy check | login message | Bob logged in. reject | login with username | Bob | and password | bad password check | login message | Bob not logged in. check not | login message | Bob logged in. ensure | login with username | Bob | and password | xyzzy note | this is a comment show | number of login attempts $symbol= | login message

So that page explains what is the fixture name and what are the method names. It looks perfectly static to me.

It would be tremendously useful to have a parser that would implement the logic described in that page. Otherwise, we have a big problem - the C# code serving the Fitnesse tests are not refactorable. This is an untenable situation for the long run and this is exactly where we find ourselves today. We want to refactor our Fitnesse C# code, but it is a mine field.

Now I realize it is possible to express the method names and/or the fixture name through a variable reference, in which case the static analysis would provide names with variable references inside. This is OK.

Does it make sense?

jediwhale commented 2 years ago

In a script table like the example shown. the class and method names can be easily seen, but this is just one of 15 table types described on the parent page http://fitnesse.org/FitNesse.FullReferenceGuide.UserGuide.WritingAcceptanceTests.SliM. And on the script table there is a dozen rules at the bottom of the page describing where class and method names will be found. As I said, this logic is implemented not in the parser but in the test execution phase.

On a positive note, a solution that doesn't handle every possible case but just processes a common style of script tables may be possible.

MarkKharitonov commented 2 years ago

Yes, an imperfect solution is better than none. The ultimate goal is to be able to refactor the code behind the Fitnesse tests. It is absolutely mandatory to be able to "compile" the Fitnesse tests to discover refactoring errors as early as possible.

The "compiler" can be written if we have a parser that can extract the names of fixtures and methods to the best of its ability. Having this parser is good enough for me, because I can then wrap it up with some simple Roslyn code to match the found fixtures and methods against the actual C# code. Of course, only where the parser is able to extract actual fixture and method names. Which could be a lot.

MarkKharitonov commented 2 years ago

If possible, I would like to revive this thread. Right now I need to decide if a certain function (GetAllDirectDepositsQuery) is used by the Fitnesse tests or not. I understand that there is no 100% correct method, but I would be happy to have a method that returns the files where this string is mentioned. I do not care if it is called or not or called through a variable reference which value is set at runtime. No, I am just interested in finding all the places it is hard coded. The problem is that being "hard coded" in fitnesse does not mean the whole string is mentioned - it could be broken up into pieces.

How can one even attempt to refactor the C# code if there is no way to check this?

jediwhale commented 2 years ago

I'm getting close to a solution...

MarkKharitonov commented 2 years ago

This is great. Thank you.

jediwhale commented 2 years ago

Added an option to analyze Slim test pages and extract method names in https://fitsharp.github.io/FitSharp/Release20220311.html

This requires a FitNesse jar from 2022-03-04 or later. Until a new FitNesse release is built, a jar is available at https://github.com/jediwhale/fitsharp/blob/master/binary/tools/fitnesse/fitnesse-20220304-standalone.jar

MarkKharitonov commented 2 years ago

This looks very promising. Will take some time to provide any feedback on this new feature, though. Thank you very much.

MarkKharitonov commented 2 years ago

Hi, I was trying to run the first command, but ran into some difficulties. Please, observe:

C:\xyz\tip2\Test\DFAcceptanceTest\FitNesse> java.exe -jar .\fitnesse-20220304-standalone.jar -c '..\DfTestRepository\AcceptanceTestSuite\FuncSuite\PayrollSuite\TaxSuite\AppSuite\EmacSuite?instruction' -b instruction.txt
Picked up _JAVA_OPTIONS: -Xmx512M
Mar 15, 2022 1:39:02 PM fitnesse.ConfigurationParameter loadProperties
INFO: No configuration file found (C:\xyz\tip2\Test\DFAcceptanceTest\FitNesse\plugins.properties)
Bootstrapping FitNesse, the fully integrated standalone wiki and acceptance testing framework.
root page: fitnesse.wiki.fs.FileSystemPage at .\FitNesseRoot#latest
logger: none
authenticator: fitnesse.authentication.PromiscuousAuthenticator
page factory: fitnesse.html.template.PageFactory
page theme: bootstrap
Unpacking new version of FitNesse resources. Please be patient...
**********************************************************
Files have been updated to a new version.
Please read the release notes on
http://localhost:9123/FitNesse.ReleaseNotes
to find out about the new features and fixes.
**********************************************************
Executing command: ..\DfTestRepository\AcceptanceTestSuite\FuncSuite\PayrollSuite\TaxSuite\AppSuite\EmacSuite?instruction
Command Output redirected to: instruction.txt
SEVERE: Error while starting the FitNesse [Cannot invoke "fitnesse.wiki.WikiPage.getHtml()" because "this.pageToRun" is null]
java.lang.NullPointerException: Cannot invoke "fitnesse.wiki.WikiPage.getHtml()" because "this.pageToRun" is null
        at fitnesse.testrunner.SuiteContentsFinder.getAllPagesToRunForThisSuite(SuiteContentsFinder.java:35)
        at fitnesse.responders.run.InstructionResponder.makeResponse(InstructionResponder.java:31)
        at fitnesse.FitNesseExpediter.createGoodResponse(FitNesseExpediter.java:145)
        at fitnesse.FitNesse.executeSingleCommand(FitNesse.java:66)
        at fitnesseMain.FitNesseMain.executeSingleCommand(FitNesseMain.java:177)
        at fitnesseMain.FitNesseMain.launch(FitNesseMain.java:129)
        at fitnesseMain.FitNesseMain.launchFitNesse(FitNesseMain.java:97)
        at fitnesseMain.FitNesseMain.launchFitNesse(FitNesseMain.java:59)
        at fitnesseMain.FitNesseMain.main(FitNesseMain.java:39)
C:\xyz\tip2\Test\DFAcceptanceTest\FitNesse>

What am I missing?

jediwhale commented 2 years ago

Separate the Wiki path with dots: java.exe -jar .\fitnesse-20220304-standalone.jar -c 'AcceptanceTestSuite.FuncSuite.PayrollSuite.TaxSuite.AppSuite.EmacSuite?instruction' -b instruction.txt

MarkKharitonov commented 2 years ago

Still no dice:

C:\xyz\tip2\Test\DFAcceptanceTest\DfTestRepository> java -jar ..\FitNesse\fitnesse-20220304-standalone.jar -c 'AcceptanceTestSuite.FuncSuite.PayrollSuite.TaxSuite.AppSuite.EmacSuite?instruction' -b instruction.txt
Picked up _JAVA_OPTIONS: -Xmx512M
Mar 16, 2022 4:34:13 PM fitnesse.ConfigurationParameter loadProperties
INFO: No configuration file found (C:\xyz\tip2\Test\DFAcceptanceTest\DfTestRepository\plugins.properties)
Bootstrapping FitNesse, the fully integrated standalone wiki and acceptance testing framework.
root page: fitnesse.wiki.fs.WikiFilePage: FitNesseRoot
logger: none
authenticator: fitnesse.authentication.PromiscuousAuthenticator
page factory: fitnesse.html.template.PageFactory
page theme: bootstrap
Unpacking new version of FitNesse resources. Please be patient...
**********************************************************
Files have been updated to a new version.
Please read the release notes on
http://localhost:9123/FitNesse.ReleaseNotes
to find out about the new features and fixes.
**********************************************************
Executing command: AcceptanceTestSuite.FuncSuite.PayrollSuite.TaxSuite.AppSuite.EmacSuite?instruction
Command Output redirected to: instruction.txt
SEVERE: Error while starting the FitNesse [Cannot invoke "fitnesse.wiki.WikiPage.getHtml()" because "this.pageToRun" is null]
java.lang.NullPointerException: Cannot invoke "fitnesse.wiki.WikiPage.getHtml()" because "this.pageToRun" is null
        at fitnesse.testrunner.SuiteContentsFinder.getAllPagesToRunForThisSuite(SuiteContentsFinder.java:35)
        at fitnesse.responders.run.InstructionResponder.makeResponse(InstructionResponder.java:31)
        at fitnesse.FitNesseExpediter.createGoodResponse(FitNesseExpediter.java:145)
        at fitnesse.FitNesse.executeSingleCommand(FitNesse.java:66)
        at fitnesseMain.FitNesseMain.executeSingleCommand(FitNesseMain.java:177)
        at fitnesseMain.FitNesseMain.launch(FitNesseMain.java:129)
        at fitnesseMain.FitNesseMain.launchFitNesse(FitNesseMain.java:97)
        at fitnesseMain.FitNesseMain.launchFitNesse(FitNesseMain.java:59)
        at fitnesseMain.FitNesseMain.main(FitNesseMain.java:39)
C:\xyz\tip2\Test\DFAcceptanceTest\DfTestRepository>

I must be doing something wrong.

jediwhale commented 2 years ago

Something wrong with the path to the suite page - the page is not found. You can run the FitNesse web server and navigate to the page in your browser and see what the URL is e.g. localhost:8008/MyTopSuite.MyBestSuite, then copy everything after the / into the -c comand e.g. -c 'MyTopSuite.MyBestSuite?instruction'

MarkKharitonov commented 2 years ago

Thanks Mike. We managed to generate the instructions file, but had difficulties running the analyzer. We are currently busy wrapping up the release, but we will definitely get back to it in a week or two and post our feedback here.

awangdf commented 2 years ago

Hello Mike,

I am someone assisting Mark with using the new finding method calls through file analysis that you provided up above. Currently we are experiencing an issue with the analyze part as a lot of our tests have table structures within tables. For example:

|query:Sample Query|$context|$ee |$start |$end | |Name |Earn |Deductions | |Name1 |1000.00 |!{!-Tax 1-!:14.50,!-Tax 2-!:92.02,!-Tax 3-!:62.00,!-Tax 4-!:35.00}|

And the instruction.txt generated from that content.txt cannot be analyzed as we get this error: System.InvalidOperationException: Leaf node has no branches. at fitSharp.Slim.Model.SlimLeaf.get_Branches() at fitSharp.Slim.Analysis.Report.ReportInstance.Detail(String pageName, String commands) at fitSharp.Machine.Model.Break2.Process(IEnumerable1 input) at fitSharp.Slim.Analysis.Report.RunReport(ApplicationUnderTest applicationUnderTest, TextReader reader, TextWriter writer) at fitSharp.Slim.Analysis.Report.Run(IList`1 commandLineArguments, Memory memory, ProgressReporter reporter) at fitSharp.Machine.Application.Shell.Run(Memory memory) at fitSharp.Machine.Application.Shell.RunInCurrentDomain(Memory memory) at fitSharp.Machine.Application.Shell.Run()

This error is not present if we remove the nested table in the sample test but this style of nesting is used in many of our tests so it is not really a viable solution to get around the problem.

Please take a look when you can, thank you!

jediwhale commented 2 years ago

Found the bug - should have a fix soon

jediwhale commented 2 years ago

fixed in fitsharp 2022.04.14 and requires latest fitnesse jar

jediwhale commented 2 years ago

Latest fitnesse jar here https://github.com/unclebob/fitnesse/actions/runs/2164097567 in "artifacts" section

MarkKharitonov commented 2 years ago

Thanks Mike. @awangdf's team will give it a try.

awangdf commented 2 years ago

Looks good now, thank you!

I will test with more intricate suites today.