[xharness] Report failure, no HTML report, test log has no failure, XML has one

spouliot commented 4 years ago

master

we should not have to read the XML to find errors

https://jenkins.mono-project.com/job/xamarin-macios-pr-builder/13199/Test_20Report/

spouliot commented 4 years ago

<test-case id="1118" name="TestNSUrlSessionHandlerCookieContainer" fullname="MonoTests.System.Net.Http.MessageHandlerTest.TestNSUrlSessionHandlerCookieContainer" result="Failed" label="Failed" time="00:00:14.9140000" asserts="4">
<failure>
<message>
<![CDATA[
Cookies Expected string length 148 but was 54. Strings differ at index 0. Expected: "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<bo..." But was: "{\n "cookies": {\n "cookie": "chocolate-chip"\n }\n}\n" -----------^
]]>
</message>
<stack-trace>
<![CDATA[
at MonoTests.System.Net.Http.MessageHandlerTest.TestNSUrlSessionHandlerCookieContainer () [0x000c9] in /Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/monotouch-test/System.Net.Http/MessageHandlers.cs:165 at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&) at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in /Library/Frameworks/Xamarin.iOS.framework/Versions/Current/src/Xamarin.iOS/mcs/class/corlib/System.Reflection/RuntimeMethodInfo.cs:395
]]>
</stack-trace>
</failure>

rolfbjarne commented 4 years ago

This isn't fixed: https://jenkins.mono-project.com/job/xamarin-macios-pr-builder/13333/Test_20Report/

This is the failing test:

<test-case id="1119" name="TestNSurlSessionHandlerCookieContainerSetCookie" fullname="MonoTests.System.Net.Http.MessageHandlerTest.TestNSurlSessionHandlerCookieContainerSetCookie" result="Failed" label="Failed" time="00:00:30.0620000" asserts="2">

The test log has this:

MonoTests.System.Net.Http.MessageHandlerTest
    [PASS] AcceptSslCertificatesServicePointManager(System.Net.Http.HttpClientHandler)
    [PASS] AcceptSslCertificatesServicePointManager(System.Net.Http.NSUrlSessionHandler)
MonoTests.System.Net.Http.MessageHandlerTest 00:00:33.3400000 ms

Which is interesting because the MessageHandlerTest has much more than 2 tests:

$ grep "[[]Test[]C]" ./tests/monotouch-test/System.Net.Http/MessageHandlers.cs -A 3
        [Test]
#if !__WATCHOS__
        [TestCase (typeof (HttpClientHandler))]
        [TestCase (typeof (CFNetworkHandler))]
#endif
        [TestCase (typeof (NSUrlSessionHandler))]
        public void DnsFailure (Type handlerType)
        {
            TestRuntime.AssertSystemVersion (PlatformName.MacOSX, 10, 9, throwIfOtherPlatform: false);
--
        [Test]
        public void TestNSUrlSessionHandlerCookies ()
        {
            var managedCookieResult = false;
--
        [Test]
        public void TestNSUrlSessionHandlerCookieContainer ()
        {
            var url = NetworkResources.Httpbin.CookiesUrl;
--
        [Test]
        public void TestNSurlSessionHandlerCookieContainerSetCookie ()
        {
            var url = NetworkResources.Httpbin.GetSetCookieUrl ("cookie", "chocolate-chip");
--
        [Test]
        public void TestNSUrlSessionDefaultDisabledCookies ()
        {
            // simple test. send a request with a set-cookie url, get the data
--
        [Test]
        public void TestNSUrlSessionDefaultDisableCookiesWithManagedContainer ()
        {
            // simple test. send a request with a set-cookie url, get the data
--
        [Test]
        public void TestNSUrlSessionEphemeralDisabledCookies ()
        {
            // assert we do throw an exception with ephmeral configs.
--
        [TestCase (typeof (HttpClientHandler))]
        [TestCase (typeof (CFNetworkHandler))]
#endif
        [TestCase (typeof (NSUrlSessionHandler))]
        public void RedirectionWithAuthorizationHeaders (Type handlerType)
        {

--
        [TestCase (typeof (HttpClientHandler))]
#endif
        [TestCase (typeof (NSUrlSessionHandler))]
        public void RejectSslCertificatesServicePointManager (Type handlerType)
        {
            TestRuntime.AssertSystemVersion (PlatformName.MacOSX, 10, 9, throwIfOtherPlatform: false);
--
        [TestCase (typeof (HttpClientHandler))]
#endif
        [TestCase (typeof (NSUrlSessionHandler))]
        public void AcceptSslCertificatesServicePointManager (Type handlerType)
        {
            TestRuntime.AssertSystemVersion (PlatformName.MacOSX, 10, 9, throwIfOtherPlatform: false);
--
        [Test]
        public void AssertDefaultValuesNSUrlSessionHandler ()
        {
            using (var handler = new NSUrlSessionHandler ()) {

xamarin / xamarin-macios

[xharness] Report failure, no HTML report, test log has no failure, XML has one #8214