search

krzema12 / kotlin-python

Python target for the Kotlin Programming Language. See https://github.com/krzema12/kotlin-python/tree/python-backend/python
https://discuss.kotlinlang.org/t/idea-python-backend/19852
48 stars 1 forks source link

Append data used to generate git history plot #41

Closed krzema12 closed 2 years ago

krzema12 commented 2 years ago

The goal is to solve the mystery of various little changes when generating the plot on different machines. The input data should give us some clue if the problem lies in the input data, or lets-plot's non-determinism.

krzema12 commented 2 years ago

@SerVB could you check out the branch from this PR and regenerate the graph on your machine? I'm curious what diff you will get for the SVG.

SerVB commented 2 years ago

Turns out it's not only the dates that are changed for me, but also some IDs:

Index: python/experiments/git-history-plot.svg
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/python/experiments/git-history-plot.svg b/python/experiments/git-history-plot.svg
--- a/python/experiments/git-history-plot.svg   (revision 5ea03585d0def4bf51ad2f675186d7710f577707)
+++ b/python/experiments/git-history-plot.svg   (date 1634385615223)
@@ -276,9 +276,9 @@
         <line x1="0.0" y1="0.0" x2="0.0" y2="451.0" stroke-width="1.0" stroke="rgb(0,0,0)" stroke-opacity="1.0">
         </line>
       </g>
-      <g transform="translate(42.5 0.0 ) " clip-path="url(#2tgkKT)" clip-bounds-jfx="[rect (0.0, 0.0), (805.9503819959181, 451.0)]">
+      <g transform="translate(42.5 0.0 ) " clip-path="url(#U1ytJI)" clip-bounds-jfx="[rect (0.0, 0.0), (805.9503819959181, 451.0)]">
         <defs>
-          <clipPath id="2tgkKT">
+          <clipPath id="U1ytJI">
             <rect x="0.0" y="0.0" width="805.9503819959181" height="451.0">
             </rect>
           </clipPath>
@@ -292,9 +292,9 @@
           </path>
         </g>
       </g>
-      <g transform="translate(42.5 0.0 ) " clip-path="url(#XxcaW9)" clip-bounds-jfx="[rect (0.0, 0.0), (805.9503819959181, 451.0)]">
+      <g transform="translate(42.5 0.0 ) " clip-path="url(#WZG3zX)" clip-bounds-jfx="[rect (0.0, 0.0), (805.9503819959181, 451.0)]">
         <defs>
-          <clipPath id="XxcaW9">
+          <clipPath id="WZG3zX">
             <rect x="0.0" y="0.0" width="805.9503819959181" height="451.0">
             </rect>
           </clipPath>
@@ -416,21 +416,19 @@
   </g>
 </svg>
 <!--
-Data used to generate the graph:
-
-DataPoint(date=Mon Nov 23 22:29:16 CET 2020, testsAll=5368, testsPassed=938)
-DataPoint(date=Thu Nov 26 09:28:49 CET 2020, testsAll=5368, testsPassed=938)
-DataPoint(date=Sat Nov 28 19:51:53 CET 2020, testsAll=5368, testsPassed=966)
-DataPoint(date=Mon Nov 30 20:19:39 CET 2020, testsAll=5368, testsPassed=1041)
-DataPoint(date=Fri Jul 16 20:24:30 CEST 2021, testsAll=5787, testsPassed=1060)
-DataPoint(date=Sat Jul 17 22:27:24 CEST 2021, testsAll=5787, testsPassed=1077)
-DataPoint(date=Sat Jul 24 10:45:48 CEST 2021, testsAll=5787, testsPassed=1196)
-DataPoint(date=Tue Aug 10 07:32:05 CEST 2021, testsAll=5787, testsPassed=1313)
-DataPoint(date=Sat Aug 14 15:25:22 CEST 2021, testsAll=5787, testsPassed=1365)
-DataPoint(date=Sat Aug 21 15:27:29 CEST 2021, testsAll=5787, testsPassed=1454)
-DataPoint(date=Mon Aug 23 11:25:15 CEST 2021, testsAll=5787, testsPassed=1455)
-DataPoint(date=Fri Sep 03 23:26:54 CEST 2021, testsAll=5787, testsPassed=1684)
-DataPoint(date=Tue Sep 07 18:20:20 CEST 2021, testsAll=5787, testsPassed=1704)
-DataPoint(date=Wed Sep 15 19:08:38 CEST 2021, testsAll=5787, testsPassed=1768)
-DataPoint(date=Thu Sep 16 22:26:11 CEST 2021, testsAll=5787, testsPassed=1936)
--->
+DataPoint(date=Tue Nov 24 00:29:16 MSK 2020, testsAll=5368, testsPassed=938)
+DataPoint(date=Thu Nov 26 11:28:49 MSK 2020, testsAll=5368, testsPassed=938)
+DataPoint(date=Sat Nov 28 21:51:53 MSK 2020, testsAll=5368, testsPassed=966)
+DataPoint(date=Mon Nov 30 22:19:39 MSK 2020, testsAll=5368, testsPassed=1041)
+DataPoint(date=Fri Jul 16 21:24:30 MSK 2021, testsAll=5787, testsPassed=1060)
+DataPoint(date=Sat Jul 17 23:27:24 MSK 2021, testsAll=5787, testsPassed=1077)
+DataPoint(date=Sat Jul 24 11:45:48 MSK 2021, testsAll=5787, testsPassed=1196)
+DataPoint(date=Tue Aug 10 08:32:05 MSK 2021, testsAll=5787, testsPassed=1313)
+DataPoint(date=Sat Aug 14 16:25:22 MSK 2021, testsAll=5787, testsPassed=1365)
+DataPoint(date=Sat Aug 21 16:27:29 MSK 2021, testsAll=5787, testsPassed=1454)
+DataPoint(date=Mon Aug 23 12:25:15 MSK 2021, testsAll=5787, testsPassed=1455)
+DataPoint(date=Sat Sep 04 00:26:54 MSK 2021, testsAll=5787, testsPassed=1684)
+DataPoint(date=Tue Sep 07 19:20:20 MSK 2021, testsAll=5787, testsPassed=1704)
+DataPoint(date=Wed Sep 15 20:08:38 MSK 2021, testsAll=5787, testsPassed=1768)
+DataPoint(date=Thu Sep 16 23:26:11 MSK 2021, testsAll=5787, testsPassed=1936)
+-->
\ No newline at end of file

By the way, python/experiments/generate-box-tests-reports.main.kts sometimes fails for me now with the following (but almost always after the second try it works):

java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
        at java.nio.CharBuffer.allocate(CharBuffer.java:335)
        at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:795)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1130)
        at org.eclipse.jgit.util.RawParseUtils.decodeNoFallback(RawParseUtils.java:1073)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1033)
        at org.eclipse.jgit.util.RawParseUtils.parsePersonIdent(RawParseUtils.java:853)
        at org.eclipse.jgit.revwalk.RevCommit.getAuthorIdent(RevCommit.java:409)
        at Generate_box_tests_reports_main.generateGitHistoryPlot(generate-box-tests-reports.main.kts:241)
        at Generate_box_tests_reports_main.<init>(generate-box-tests-reports.main.kts:46)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at kotlin.script.experimental.jvm.BasicJvmScriptEvaluator.evalWithConfigAndOtherScriptsResults(BasicJvmScriptEvaluator.kt:96)
        at kotlin.script.experimental.jvm.BasicJvmScriptEvaluator.invoke$suspendImpl(BasicJvmScriptEvaluator.kt:41)
        at kotlin.script.experimental.jvm.BasicJvmScriptEvaluator.invoke(BasicJvmScriptEvaluator.kt)
        at org.jetbrains.kotlin.scripting.compiler.plugin.AbstractScriptEvaluationExtension$doEval$1.invokeSuspend(AbstractScriptEvaluationExtension.kt:143)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at org.jetbrains.kotlin.scripting.compiler.plugin.AbstractScriptEvaluationExtension.doEval(AbstractScriptEvaluationExtension.kt:135)
        at org.jetbrains.kotlin.scripting.compiler.plugin.AbstractScriptEvaluationExtension.eval(AbstractScriptEvaluationExtension.kt:123)
        at org.jetbrains.kotlin.cli.jvm.K2JVMCompiler.doExecute(K2JVMCompiler.kt:102)
        at org.jetbrains.kotlin.cli.jvm.K2JVMCompiler.doExecute(K2JVMCompiler.kt:52)
        at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.kt:92)
        at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.kt:44)
krzema12 commented 2 years ago

Thanks for your diff, I'll work on printing the dates in such way that they are timezone-agnostic (probably in UTC). I'm aware of changing IDs, that's why I ignore them by removing them in GitHub Actions :) Check the code that you commented out yesterday.

About the OutOfMemory, it never happened to me. I assume it's because of analyzing huge git history of Kotlin repo and the stack trace points to it. However, what's weird for me is that it fails on your machine which is much stronger than mine. Maybe it depends on memory usage at the moment of running the script. Anyway, thanks for the info, it's something to keep an eye on. If it works with the second try, I assume we can shelve it for now.

krzema12 commented 2 years ago

I'd love to understand and fix it, but I don't have time :(