TestEpochWrite and TestEpochParse run failed

[tpolong]

TestGeoJsonSerDe and TestEsriJsonSerDe both can't run.

[randallwhitman]

Hi @tpolong - thanks for reporting this. I'm curious in what time zone was that run - in case it makes a difference.

[tpolong]

I'm in the 8th East District

[randallwhitman]

Sorry, I'm not sure what time-zone that is — please see - https://24timezones.com/time-zones

[tpolong]

UTC+8

[randallwhitman]

Oh, "can't run" may mean a compilation problem rather than failed test - i may have misread it. (the image does not mean much to me.)

[tpolong]

您好，您的邮件已收到，谢谢！

[randallwhitman]

~Even worse, the compile error is in BaseJsonSerde, in the product source itself. spatial-framework-for-hadoop/hive/src/main/java/com/esri/hadoop/hive/serde/BaseJsonSerDe.java:[324,24] error: method getPrimitiveEpoch in class HiveShims cannot be applied to given types; Indicates that it may have broken after testing when cleaning up for check-in, and that the binaries published for v2.2.0 may not match the source labeled with v2.2.0 tag.~ [No, the compile error I saw Friday was due to an unfinished change in my working directory, not in v2.2.0 itself.]

[randallwhitman]

Both TestEsriJsonSerDe and TestGeoJsonSerDe just passed for me. I'm scheduled tests at varying times today, so I can see of the results vary by time of day (e.g. around 00:00 UTC), if in fact mvn test will run as scheduled by at(1).

[randallwhitman]

Cross-reference #141 #165

[randallwhitman]

All tests passed for me, at all the following times, PST: 0030 0230 0330 0345 0400 0415 0430 0530 1045 1130 1230 1330 1430 1530 1545 1600 1615 1630 1830 2030 2230 I would need additional information in order to try to work any on this (details in text, not image only; and explanation in English/Spanish).

[tpolong]

---

Test set: com.esri.hadoop.hive.serde.TestGeoJsonSerDe

Tests run: 12, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.108 sec <<< FAILURE! TestEpochParse(com.esri.hadoop.hive.serde.TestGeoJsonSerDe) Time elapsed: 0 sec <<< FAILURE! java.lang.AssertionError: expected:<1703> but was:<1702> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at com.esri.hadoop.hive.serde.TestGeoJsonSerDe.TestEpochParse(TestGeoJsonSerDe.java:168) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) TestEpochWrite(com.esri.hadoop.hive.serde.TestGeoJsonSerDe) Time elapsed: 0.001 sec <<< FAILURE! java.lang.AssertionError: expected:<1703> but was:<1702> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at com.esri.hadoop.hive.serde.TestGeoJsonSerDe.TestEpochWrite(TestGeoJsonSerDe.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

[tpolong]

---

Test set: com.esri.hadoop.hive.serde.TestEsriJsonSerDe

Tests run: 15, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.625 sec <<< FAILURE! TestEpochParse(com.esri.hadoop.hive.serde.TestEsriJsonSerDe) Time elapsed: 0.005 sec <<< FAILURE! java.lang.AssertionError: expected:<1703> but was:<1702> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at com.esri.hadoop.hive.serde.TestEsriJsonSerDe.TestEpochParse(TestEsriJsonSerDe.java:201) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) TestEpochWrite(com.esri.hadoop.hive.serde.TestEsriJsonSerDe) Time elapsed: 0 sec <<< FAILURE! java.lang.AssertionError: expected:<1703> but was:<1702> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at com.esri.hadoop.hive.serde.TestEsriJsonSerDe.TestEpochWrite(TestEsriJsonSerDe.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

[tpolong]

Failed tests: TestEpochParse(com.esri.hadoop.hive.serde.TestEsriJsonSerDe): expected:<1703> but was:<1702> TestEpochWrite(com.esri.hadoop.hive.serde.TestEsriJsonSerDe): expected:<1703> but was:<1702> TestEpochParse(com.esri.hadoop.hive.serde.TestGeoJsonSerDe): expected:<1703> but was:<1702> TestEpochWrite(com.esri.hadoop.hive.serde.TestGeoJsonSerDe): expected:<1703> but was:<1702> Tests run: 49, Failures: 4, Errors: 0, Skipped: 0

[randallwhitman]

Thanks for the test details. This is curious, why we see different results. Did you test with default versions of Hive etc (that is, no -P arguments to mvn)? I'm guessing this has something to do with time zones and/or HIVE-12192: Hive should carry out timestamp computations in UTC.

[randallwhitman]

Noticing that the test results were posted about 23:30 PST, I've scheduled tests to run here at 23:30 UTC-8, to see if at that time of day it will repro on a computer in PST. I'm curious if it will repro in both morning and afternoon/evening in UTC+8.

[tpolong]

[tpolong]

[randallwhitman]

I think that is saying that it happens with Hive-3.1? But does not mention what time of day the test was run? (Here in PST additional test runs passed at 2315 2330 2345 0015.)

[tpolong]

System.out.print(new Date().getTime()); 1640766137444

[randallwhitman]

I think the next thing we want to figure out, is, whether the issue is with the tests only, versus whether the issue is in the Spatial-Framework product code itself.

[randallwhitman]

In a VM set to UTC+0930 I repro 4 tests failed.

[randallwhitman]

TestEpochParse - epochFromWritable(fieldData) gives 1974-08-30 00:00 UTC when run in East TZ but 1974-08-31 00:00 UTC when run in West TZ, -Phive-3.1 -Phadoop-3.2 Maybe we're getting something unexpected from the Hive API.

[randallwhitman]

I'm thinking to changing dependency defaults to up-to-date versions, then afterward circle back to whether to try to work around HIVE-12192 for Hive-3.0 and below.

[randallwhitman]

Is the issue blocking analysis? If a matter of build, the JAR file can be built with -DskipTests=true.

[tpolong]

I have tested the latest code and it works fine

[randallwhitman]

Thanks! I went ahead and merged the PR.