CoreCLR support for FsCheck.Xunit

josephdecock commented 8 years ago

Is it possible to add support for the Core CLR to FsCheck.Xunit? I'm getting "The dependency FsCheck.Xunit 2.2.4 ... does not support framework DNXCore,Version=v5.0." I would prefer to send a pull request rather than just opening an issue, and I have cloned the repo and poked around a bit. Unfortunately I'm still learning paket and the rest of the tools in the build, and what to do was not immediately obvious. Can I get a nudge in the right direction?

As far as I can tell, neither FsCheck nor Xunit alone have any trouble running on the Core CLR, so I would assume that there isn't anything fundamental in the way of this happening. Is that true?

mausch commented 8 years ago

Last I heard ( https://github.com/SuaveIO/suave/pull/334 ) "the [F#] compiler is very ... very alpha quality"

kurtschelfthout commented 8 years ago

Interesting, are you saying you actually ran FsCheck as is on Core CLR and it works? Because afaik that's never been tested.

As @mausch says, I was expecting to wait with this until the F# compiler is not quite so alpha. So I feel like the nudge would be more like a shove down the rabbit hole :)

josephdecock commented 8 years ago

Wow, I seem to have stumbled into much deeper waters than I intended. Hopefully I'm not about to take the blue pill and wake up with wires coming out of my brain...

Anyway, what I've done with FsCheck on the Core CLR is pretty minimal. I followed this guide to getting started with xunit using dnx. Then I installed FsCheck and was able to write and compile a test that included a call to Prop.ForAll. The output of that test didn't quite make sense, but that may have been my fault. I'm just getting into FsCheck and so far have used it with Xunit. I hoped things would make more sense with FsCheck.Xunit and tried installing it, and that brought me to the dependency errors.

josephdecock commented 8 years ago

I've cleaned up the tests that I made, and FsCheck definitely works when targeting the CoreCLR with C#. I have a demo here.

kurtschelfthout commented 8 years ago

That's pretty cool!

I don't know enough about CoreCLR to really understand things. I wonder why FsCheck.Xunit complains but FsCheck itself does not; they are compiled in the same way in the same MSBuild step. It must have something to do with the xunit dependency, which does have different versions for CoreCLR afaik.

So I'm currently not in a position to give you any nudges :( Happy to answer any questions though. I'll try to do some research on CoreCLR to see what can reasonably be done here, might take a bit of time.

jhamm commented 8 years ago

Thanks for getting the demo setup. It was very helpful!

To get dnvm, dnu, and dnx commands, follow the instructions: https://github.com/aspnet/home

To run the demo

> dnvm install 1.0.0-rc1-update1 -r coreclr -a x86

> dnvm list

Active Version           Runtime Architecture OperatingSystem Alias
------ -------           ------- ------------ --------------- -----
  *    1.0.0-rc1-update1 coreclr x86          win

> cd {project folder}/Point
> dnu restore
> dnu build
> dnx test

To see the error

in the Point/project.json, add to the dependencies; "FsCheck.Xunit": "2.2.4", Add the using "FsCheck.Xunit" Add a sample test like:

        [Property]
        public bool SameCoordsShouldBeEqual(int x, int y)
        {
            var p1 = new Point(x, y);
            var p2 = new Point(x, y);
            return p1 == p2;
        }

Then run:

> dnu restore
> dnu build
> dnx test

jhamm commented 8 years ago

Here is a nudge in the right direction,

Have a look at the "PCL" folder in FsCheck solution, ex: FsCheck.Portable.Profile7. Each of the files in the projects are links to the non-portable FsCheck file so if one changes the portable will also use the same file.

You will need to add 2 references that the non-portable FsCheck.Xunit project uses, FsCheck and xunit.execution.desktop (need to use paket add for this one).

Take a look at the src\FsCheck\paket.template file to see how to add the portable versions to the nuget package.

The build.fsx is how Paket will Pack the nuget file for a new release.

Let me know if you need another nudge :smile:

kurtschelfthout commented 8 years ago

Here's an interesting tutorial on migrating F# to CoreCLR.

https://github.com/enricosada/fsharp-dotnet-cli-samples/wiki/Migrate-fsproj-To-.NET-CLI

josephdecock commented 8 years ago

Thanks for all the input, especially @jhamm. I've had some partial success. I've created a Portable Profile for FsCheck.Xunit, and added the resulting dll to the nuget package. Using that package in my demo project, I'm able to run tests created with the PropertyAttribute! The code is here.

As I said, this was only a partial success however. The duration for tests created with the Property Attribute are missing from the test runner. Any idea why that's the case? In case that's unclear, here's what I'm looking at in the test runner: The one test missing duration info is the one I've defined as a [Property]. The rest are my old tests defined as [Fact]s. For a while I thought maybe the test wasn't getting run at all, but I am able to debug the test and step through the code in the property.

I also wonder if creating more pcl profiles is the right approach. To get started, I just created one (FsCheck.Xunit.Portable.Profile7). Would you want Profile78 and Profile259 too? Then if nunit support was ever considered, would you want 3 Profiles for that too? Seems like a lot of very similar files, but then again I don't have prior experience compiling PCLs, so maybe its just normal.

kurtschelfthout commented 8 years ago

This is really interesting and cool work.

Can someone explain the link between (existing) PCLs and CoreCLR or give a pointer? Seems like certain PCL profiles work on CoreCLR, or something?

Sorry don't directly know why the time is not showing - if you debug into the PropertyAttribute, does it return a TestPassed with a timer.Total that makes sense?

EDIT: forgot to mention, yeah don't worry about the different files. Seems to work fine and not to cause too much hassle (perhaps counter-intuitively).

josephdecock commented 8 years ago

This blog from last year discusses how PCLs and CoreCLR interact. It says that PCLs can run as-is on CoreCLR. I'm still looking for more thorough documentation, but it's a start.

When I debug into the attribute, the timer.Total doesn't make sense. I get a non-zero number of ticks less than 1 millisecond. I've got a test that takes several seconds to run (just wasting cpu cycles in a big loop), and still the timer.Total is just a few thousand ticks (.2 ms).

kurtschelfthout commented 8 years ago

Thanks for that link. Clears up things a bit. It says profile259+ runs on CoreCLR, so I would be fine if that would be the only profile for FsCheck.Xunit (i.e. we can skip 7 and 78 if we like, I put those in because FsCheck is used with the F# compiler tests which tests those two profiles as well)

The other thing it says is to put the dll in the lb/dotnet folder in NuGet, and to list dependencies explicitly. Though it seems like things work without it. May be a bit dodgy.

Timer looks like a separate bug to me. I wonder if/how this works on NET4.5. The ExecutionTimer is instantiated but never used. It looks to me like it should time the Check.Method execution. @jhamm and/or @ploeh may know a bit more about this?

kurtschelfthout commented 8 years ago

Can you try the timing now, after pulling that commit I just pushed?

josephdecock commented 8 years ago

With that commit I get the the timing I would expect. Thanks a lot!

I've been trying to get my head around the future of PCLs in Net Core. This talk from NDC London was really interesting. Damien Edwards and David Fowler talk about the new .NET CLI and .NET Platform Standard (documented here). The platform standard is the successor to PCLs and it looks like a really good direction to go in. You just say "I'm building my library against version x.y of the standard" and then any platform that implements that version or later of the standard can use it. You aren't tied down to specifying in advance which platforms can use your library.

All that said, I think for now it makes sense to wait to adopt the platform standard. Work on it is still in progress . In the talk, they said that targeting the platform standard would be the big change to come in ASP.NET 5's rc 2, which should be coming Real Soon Now(tm).

With all these big changes on the horizon, would you even want to consider a PR of what I'd done reworked as a profile259?

kurtschelfthout commented 8 years ago

Thanks for the info. I agree I don't really want to do the "dotnet" stuff, seems like they're going to get rid of it, and the platform standard seems like it's the way forward, but as you say not quite ready yet.

That said, I'd definitely be open to accept a PR that adds Profile259 support for FsCheck.Xunit. From my perspective supporting an additional profile for FsCheck.Xunit has low marginal cost. I've been pretty happy with how stable PCL's are, and reading that documentation (haven't watched the video yet) it sounds like there is a good path forward to targeting the Standard Platform from PCL. Most of the bits are already there in the build scripts etc for FsCheck anyway.

I leave it up to you. If you're worried perhaps it's worth just sending what you have without any additional work, and then we know exactly what we're talking about?

enricosada commented 8 years ago

@kurtschelfthout i can do a .NET Core build with .NET CLI if you want, what branch should i target? master or fscheck3? it must use f# 4 (no FSharp.Core 3.x on .NET Core), so it's fscheck3 the best place?

Usually it's only a project.json and some ifdef about migration .NET Full -> .NET Core.

the netstandard framework now is ok ( dotnet and dnxcore50 are migrated to netstandard ).

fscheck it's interesting because i want to try dotnet test using xunit (the only test framework supported atm) and f#. FsCheck it's used by fsharp (visualfsharp) repo so it's a dependancy anyway for a full .net core build of the compiler.

kurtschelfthout commented 8 years ago

Yeah, fscheck3 is the right branch in that case. Yes, I'm interested! Thanks.

On 23 Mar 2016, at 01:58, Enrico Sada notifications@github.com wrote:

@kurtschelfthout i can do a .NET Core build with .NET CLI if you want, what branch should i target? master or fscheck3? it must use f# 4 (no FSharp.Core 3.x on .NET Core), so it's fscheck3 the best place?

Usually it's only a project.json and some ifdef about migration .NET Full -> .NET Core.

the netstandard framework now is ok ( dotnet and dnxcore50 are migrated to netstandard ).

fscheck it's interesting because i want to try dotnet test using xunit (the only test framework supported atm) and f#. FsCheck it's used by fsharp (visualfsharp) repo so it's a dependancy anyway for a full .net core build of the compiler.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

la-yumba commented 8 years ago

I'm trying to use fsCheck with .NET Core 1.0.0 and (preferably) NUnit and get the following results:

Package dependencies:

Adding FScheck 2.5.0 dependency causes no problems
Adding FSCheck.NUnit (or .XUnit) as dependencies does not work (does not support framework .NETCoreApp); I've seen there's some WIP in this area; please include NUnit as well :-)

Behaviour:

the only way to get a failing test to fail is running the tests from the console with dotnet test, and using .QuickCheckThrowOnFailure()
if I use the test runner within VS the test always passes
if I use QuickCheck() the test always passes (even when running from the command line)

enricosada commented 8 years ago

I'll send a pr tonight. I. Converted fscheck, fscheck.xunit to dotnet core for my webcast about .net core.

kurtschelfthout commented 8 years ago

@la-yumba Yes FsCheck itself should work as one of the profiles we ship is compatible with .NET Standard Platform. We don't currently ship those for FsCheck.Xunit or Nunit.

Also without any integration (which won't work for now with .net core) you have to use QuickCheckThrowOnFailure so FsCheck throws an exception on a failing test. Simple QuickCheck will just print the result to stdout but otherwise not do anything.

enricosada commented 8 years ago

ok added new pr, with both xunit and nunit, targeting .net core. the dotnet test is supported (and used in fscheck tests)

kurtschelfthout commented 7 years ago

@enricosada's PR was merged to fscheck3 branch. Closing this issue as a result.

la-yumba commented 7 years ago

@kurtschelfthout can you please notify us on this thread when the code changes re .net Core support are published, with the relevant version of the NuGet package? Thanks,

kurtschelfthout commented 7 years ago

I'll try to remember. Meanwhile you can use the 3.0.0-alpha builds on the AppVeyor feed:

https://ci.appveyor.com/project/kurtschelfthout/fscheck/history

See readme for detailed instructions.

fscheck / FsCheck

CoreCLR support for FsCheck.Xunit #199

To run the demo

To see the error