smalltalkCI: JUnit XML Prettifier

fniephaus commented 7 years ago

Just a quick heads up that I am about to deprecate smalltalkCI's junit_xml_prettfier.py in favor of a new Smalltalk implementation (see dev) soon. The script is used by GsDevKit_home here and I will send a PR accordingly when the time has come.

dalehenrich commented 7 years ago

Oh good:)

On 09/27/2016 06:40 AM, Fabio Niephaus wrote:

Just a quick heads up that I am about to deprecate smalltalkCI's |junit_xml_prettfier.py| in favor of a new Smalltalk implementation (see |dev| https://github.com/hpi-swa/smalltalkCI/tree/dev) soon. The script is used by GsDevKit_home here https://github.com/GsDevKit/GsDevKit_home/blob/c6fe23cd4ba3ca78d2f2f1904394c4781d753287/bin/smalltalkCI#L114 and I will send a PR accordingly when the time has come.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/GsDevKit/GsDevKit_home/issues/141, or mute the thread https://github.com/notifications/unsubscribe-auth/AAmFT2kAaW0WER0I-Zb05QSw-j3QTPB5ks5quRzngaJpZM4KHrdU.

fniephaus commented 7 years ago

Hi @dalehenrich,

It looks like a project is loaded into the server image twice? See L100 vs. L111.

Is this true? What's the reasoning behind this?

dalehenrich commented 7 years ago

I would guess that I didn't have a #test:xmlLogDirPath: method to use:) It does look like a bug ... I imagine that I was more interested in controlling where the xml file was produced than I was worried about reloading a project (a reload is pretty cheep) ...

fniephaus commented 7 years ago

Ok no worries. Do you still care where the XML files are placed? Now that the images print the build report to stout, there's no need to call a Python script anymore :) But smalltalkCI still produces an XML file and puts it next to the image.

dalehenrich commented 7 years ago

Excellent ... I have nothing against python, but ... :)

I still think that we need explicitly to control where the XML file goes for GemStone ... "next to the image" is not always where you think it is in GemStone as there really isn't necessarily an "image" per se ... so for that particular script the location of the XML file is specified and we need to be able to continue to do that for GemStone.

Dale

On 09/30/2016 07:44 AM, Fabio Niephaus wrote:

Ok no worries. Do you still care where the XML files are placed? Now that the images print the build report to stout, there's no need to call a Python script anymore :) But smalltalkCI still produces an XML file and puts it next to the image.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GsDevKit/GsDevKit_home/issues/141#issuecomment-250762981, or mute the thread https://github.com/notifications/unsubscribe-auth/AAmFT8UTwceQ6iIjwRegJiezx1QPnkDmks5qvSBTgaJpZM4KHrdU.

dalehenrich commented 7 years ago

additional work needed after pr #144 to get stdout from smalltalk routed to stdout of the script ... use startTopaz

dalehenrich commented 7 years ago

@fniephaus it looks like both SmalltalkCIGemstone class>>test:xmlLogDirPath: and SmalltalkCIGemstone class>>test:projectDirectory: and maybe others are not setting suiteName which is used writing the xml file and i dumping the output from the stdout reporter so I'm guessing that answering nil for suiteName is probably causing an error and I assume that there are error handlers involved that are swallowing the error? and just skipping the results output perhaps?

Anyway, I switched to using topaz like the run.sh and still didn't see output, but then those two variants are broken with respect to suiteName...

dalehenrich commented 7 years ago

Hmmm, compare the (travis-ci output](https://travis-ci.org/GsDevKit/GsDevKit_home/jobs/166299474#L2978-L2981https://travis-ci.org/GsDevKit/GsDevKit_home/jobs/166299474#L2978-L2981) with the raw ouput log:

topaz 1>   run
    SmalltalkCIGemstone 
      test: '/home/travis/build/GsDevKit/GsDevKit_home/shared/repos/smalltalkCI/.smalltalk.ston'
      xmlLogDirPath: '/home/travis/build/GsDevKit/GsDevKit_home/server/stones/travis1/ci'
%
travis_fold:start:test_project
[0Ktravis_time:start:test_project_timer
[0K[1m[34mTesting project...[0m
Running suite "nil" with 40 tests.
Running suite "SmalltalkCI" with 15 tests.
Project 'SomeNonExistingProject' not registered (loaded)
Finished running suite: SmalltalkCI
Running suite "SmalltalkCI-Excluded" with 6 tests.
Finished running suite: SmalltalkCI-Excluded
Running suite "SmalltalkCI-Excluded" with 6 tests.
Finished running suite: SmalltalkCI-Excluded
Running suite "MyRunnerName" with 15 tests.
Project 'SomeNonExistingProject' not registered (loaded)
Finished running suite: MyRunnerName
Running suite "SmalltalkCI" with 15 tests.
Project 'SomeNonExistingProject' not registered (loaded)
Finished running suite: SmalltalkCI
Running suite "SmalltalkCI-Excluded" with 6 tests.
Finished running suite: SmalltalkCI-Excluded
Project 'SomeNonExistingProject' not registered (loaded)
Finished running suite: nil

travis_time:end:test_project_timer:start=1476060263378000000,finish=1476060264217000000,duration=839000000
[0Ktravis_fold:end:test_project
[0K

[1m[34m##################################################

[1m[34m# nil                                            #

[1m[34m# 40 Tests with 0 Failures and 0 Errors in 0.83s #

[1m[34m##################################################

[0mtravis_fold:start:passing_tests
[0K[1m[32m40 passing Tests

[0m[1m

SmalltalkCITest

[0m[32m âœ“ [0mSmalltalkCITest>>#testClassesForCategories (1ms)

[32m âœ“ [0mSmalltalkCITest>>#testClassesForPackages (4ms)

[32m âœ“ [0mSmalltalkCITest>>#testClassesFrom (4ms)

[32m âœ“ [0mSmalltalkCITest>>#testClassesInCategory (0ms)

[32m âœ“ [0mSmalltalkCITest>>#testClassesInPackage (2ms)

[32m âœ“ [0mSmalltalkCITest>>#testClassesOfProjects (4ms)

[32m âœ“ [0mSmalltalkCITest>>#testClassesToTest (12ms)

[32m âœ“ [0mSmalltalkCITest>>#testClassesWithCategoryNames (0ms)

[32m âœ“ [0mSmalltalkCITest>>#testClassesWithPackageNames (4ms)

[32m âœ“ [0mSmalltalkCITest>>#testNew (0ms)

[32m âœ“ [0mSmalltalkCITest>>#testResolveAllWith (0ms)

[32m âœ“ [0mSmalltalkCITest>>#testResolveWith (0ms)

[32m âœ“ [0mSmalltalkCITest>>#testSuiteName (0ms)

[32m âœ“ [0mSmalltalkCITest>>#testTravisDetection (0ms)

[32m âœ“ [0mSmalltalkCITest>>#testTravisFold (0ms)

[1m

SCIMetacelloLoadSpecTest

[0m[32m âœ“ [0mSCIMetacelloLoadSpecTest>>#testAuthor (0ms)

[32m âœ“ [0mSCIMetacelloLoadSpecTest>>#testIsComplete (0ms)

[32m ï¿½ï¿½ï¿½ [0mSCIMetacelloLoadSpecTest>>#testSimple (0ms)

[1m

SCIAbstractConfigTest

[0m[32m âœ“ [0mSCIAbstractConfigTest>>#testSimple (0ms)

[1m

SCIGoferLoadSpecTest

[0m[32m âœ“ [0mSCIGoferLoadSpecTest>>#testAddLoadedClassesFrom (1ms)

[32m âœ“ [0mSCIGoferLoadSpecTest>>#testIsComplete (0ms)

[32m âœ“ [0mSCIGoferLoadSpecTest>>#testSimple (0ms)

[1m

SCIGemStoneServerConfigSpecTest

[0m[32m âœ“ [0mSCIGemStoneServerConfigSpecTest>>#testSimple (0ms)

[1m

SCIMonticelloLoadSpecTest

[0m[32m âœ“ [0mSCIMonticelloLoadSpecTest>>#testAddLoadedClassesFrom (0ms)

[32m âœ“ [0mSCIMonticelloLoadSpecTest>>#testIsComplete (0ms)

[32m âœ“ [0mSCIMonticelloLoadSpecTest>>#testRepository (0ms)

[32m âœ“ [0mSCIMonticelloLoadSpecTest>>#testSimple (0ms)

[1m

SCIAbstractLoadSpecTest

[0m[32m âœ“ [0mSCIAbstractLoadSpecTest>>#testIsComplete (0ms)

[32m âœ“ [0mSCIAbstractLoadSpecTest>>#testIsPlatformCompatible (0ms)

[32m âœ“ [0mSCIAbstractLoadSpecTest>>#testloadProjectOn (0ms)

[32m âœ“ [0mSCIAbstractLoadSpecTest>>#testSimple (0ms)

[1m

SCITestReporterStdoutTest

[0m[32m âœ“ [0mSCITestReporterStdoutTest>>#testReport (69ms)

[32m âœ“ [0mSCITestReporterStdoutTest>>#testReportFailure (10ms)

[32m âœ“ [0mSCITestReporterStdoutTest>>#testReportSuccess (38ms)

[1m

SCITestRunnerTest

[0m[32m âœ“ [0mSCITestRunnerTest>>#testRunClassesNamed (43ms)

[1m

SentButNotImplementedTest

[0m[32m âœ“ [0mSentButNotImplementedTest>>#testSentButNotImplemented (623ms)

[1m

SmalltalkCISpecTest

[0m[32m âœ“ [0mSmalltalkCISpecTest>>#testAddSpecs (0ms)

[32m âœ“ [0mSmalltalkCISpecTest>>#testFromStream (0ms)

[32m âœ“ [0mSmalltalkCISpecTest>>#testSimple (0ms)

[1m

UndefinedSymbolsTest

[0m[32m âœ“ [0mUndefinedSymbolsTest>>#testUndefinedSymbols (0ms)

travis_fold:end:passing_tests
[0K

[1m[32m  Executed 40 Tests with 0 Failures and 0 Errors in 0.83s.[0m

aSmalltalkCIGemstone
topaz 1>   logout
--- 10/10/2016 00:44:24.248 UTC Logging out
topaz>   exit 0
[92;1m...finished :: startTopaz travis1 -l -T 100000[0m
[92;1m...finished :: smalltalkCI -r -z /home/travis/build/GsDevKit/GsDevKit_home/shared/repos/smalltalkCI/.smalltalk.ston travis1[0m

travis_time:end:2630f0db:start=1476059951720397623,finish=1476060264761232395,duration=313040834772
[0K
[32;1mThe command "tests/testTravisCI.sh" exited with 0.[0m

Done. Your build exited with 0.

With my change to using startTopaz the output is being dumped correctly, but apparently the travis_fold:start:test_project line immediately following the % on the 6th line is obscuring the rest of the output ... from travis ... I don't see a closing travis fold statement --- could that be the cause of the missing output? Presumably run on the command line, interactively the output will show up correctly? I'm not sure what the travis_fold does and what happens when there is no function defined (i.e., dumping output in a non-travis environment) ... I will go ahead and merge commit fb698c0, but I will let you figure out the 1travis_fold` issue ...

fniephaus commented 7 years ago

@fniephaus it looks like both SmalltalkCIGemstone class>>test:xmlLogDirPath: and SmalltalkCIGemstone class>>test:projectDirectory: and maybe others are not setting suiteName which is used writing the xml file and i dumping the output from the stdout reporter so I'm guessing that answering nil for suiteName is probably causing an error and I assume that there are error handlers involved that are swallowing the error? and just skipping the results output perhaps?

I have changed the behavior of how suiteName works a little bit in one of the last few commits, but I think I know what's necessary to fix this for you again.

Also, travis_fold:end:test_project is present in the log you provided, even a few times. However, the Travis UI doesn't display the log correctly, because the identifiers (in this example test_project) are not unique and that breaks it. I'm on it...

Thanks for your work, I should be able to fix the rest by updating smalltalkCI 😃

dalehenrich commented 7 years ago

Thanks, man! ... the pharo-3.0 issues seem to be intermittent as I haven't seen the bad behaviour pop up recently ...

fniephaus commented 7 years ago

Are you using a Pharo-3 client somewhere in GsDevKit_home's tests?

dalehenrich commented 7 years ago

Yes pharo-3 is used in just about every test, but I specifically use smalltalkCI here ... not for running client tests but for loading code into the pharo-3 client ... and then of course tests are run as part of smalltalkCI and I suppose it's the running tests in clients from GemStone that must be broken

dalehenrich commented 7 years ago

@fniephaus as I look closer at the pharo-30 failure ... I am noticing understading the problem is complicated by the fact that the travis ci output cuts off with the pharo4.0 load, so you have to look at the raw log to figure things out ...

If you search for createClient -t pharo travisClient_Pharo3.0 you will find that that a pharo3 client was created and that the image was successfully created ... and somehow disappeared or was deleted(?) between the time the image was created and the time an attempt was made to start the image!

fniephaus commented 7 years ago

I was busy trying to fix GsDevKit yesterday and unfortunately havent had the time to fix the Travis folds yet. And yes, I was also looking at the raw output. I'm not sure if it actually was deleted or if it was never there. That's what I currently need to find out...

dalehenrich commented 7 years ago

The last master branch pass was this build 20 hours ago and the pharo3.0 client passed ... so something in smalltalkCI changed in the last 20 hours to cause pharo-30 client test to fail ...

dalehenrich commented 7 years ago

... and the changes to GsDevKit_home are all yours as well ... although I'm not quite sure what changes actually ended being kept ...

fniephaus commented 7 years ago

I can only imagine that it was this commit then. smalltalkCI saves the image after loading now...

dalehenrich commented 7 years ago

these are the only changes that survived ... and the commit you are referring to is not used by GsDevKit_home ...

dalehenrich commented 7 years ago

I think that this is the code that does the client image save and it is doing a save ... the save command is used for all of the pharo-3.0 image creation and works fine for the creation of tode clients ..

we're back to analyzing the changes that were made to the gemstone image code as well as scripts in the last 20 hours ... perhaps there is some image cleanup code in smalltalk now that might be tossing the image?

fniephaus commented 7 years ago

I'm currently not at home, but I will have a look as soon as I am. Yes, that's the line I also thought would save the image correctly and independently from smalltalkCI. And no, smalltalkCI is definitely not removing images, but I restarted this build yesterday to see if the problem is in GsDevKit. The commit is from end of August, so I am afraid there's a lot more to look at than you thought...

dalehenrich commented 7 years ago

... haha ... I was going by the hours since the test was run :) ... in scanning back in the builds to find the last passed master test I didn't see too many passing tests so it would be tedious to work your way back though the test results to isolate the commit where the client tests started failing ... but that may be worth trying?

fniephaus commented 7 years ago

Mh, I often even didn't run the GemStone tests because I canceled builds as soon as they were failing. I'm fixing the folding issue now and then the suiteName bug. The Pharo-3 client might need to wait until tomorrow...

dalehenrich commented 7 years ago

okay .. no problem ... this is not holding me up at the moment ...

fniephaus commented 7 years ago

That's good, but I want to get it working again asap. I've just pushed fixes for the Travis folds and suiteName...so that should be working now.

dalehenrich commented 7 years ago

Somehow I was able to get a version of the master branch of SmalltalkCI that is getting compile errors while loading into GemStone ...

dalehenrich commented 7 years ago

Yeah this commit ... to the master branch does not compile fo GemStone ... are you just changing code on the master branch without testing?

dalehenrich commented 7 years ago

You really shouldn't be changing code on the master branch without any testing ... I said no problem because at least non-client builds were working on travis ... now smalltalkCi is just plain broken for GemStone ... not cool ... I am trying to get build results specifically for GemStone for the Metacello project ... and now builds are broken because of smaltalkCI errors ...

dalehenrich commented 7 years ago

@fniephaus pinging you directly in case you've missed the fact that you've broken smalltalkCI for GemStone ... you should really run tests on a separate branch and get out of the habit of doing development on the master branch ... not very friendly to your users!

fniephaus commented 7 years ago

Yes, indeed. That change broke it. I've tested several Squeak and Pharo images locally and I was just monitoring the master build for failing GemStone tests, but you were faster. Testing on dev adds +30mins that I just don't have tonight.

dalehenrich commented 7 years ago

Yeah but you just cost me an hour ....

dalehenrich commented 7 years ago

There is never a good reason to do development on the master branch!

dalehenrich commented 7 years ago

@fniephaus ... when I said "okay .. no problem ... this is not holding me up at the moment ..." ... I meant ... there is no reason to hurry and push broken code to the master branch, because I can live with the brokenness that you already introduced by doing development on the master branch ... but instead you've broken the master branch even more ...

this is not acceptable

fniephaus commented 7 years ago

I really didn't mean to waste your time, instead I was just trying to make your and my life easier. Unfortunately, I have lots of other things to do this week, and I wasn't able to finish everything I wanted to finish yesterday. So I was sacrificing my evenings to make up for that. Anyhow, you were right that I wasn't following best practices. That's why I have just re-enable branch protection on master which forces me to Travis CI check everything and then open PRs again.

fniephaus commented 7 years ago

Ah, a fix for GemStone has already been deployed 25mins ago...

dalehenrich commented 7 years ago

If your development had all taken place on a development branch there would be no need to sacrifice your evenings patching broken master branches ...

I'm glad that you are returning to best practices ... they are best practices for a reason:)

dalehenrich commented 7 years ago

After all of this the client builds are still broken :(

fniephaus commented 7 years ago

Are you referring to Pharo-3? I'm afraid that needs more time than I have tonight (as mentioned in https://github.com/GsDevKit/GsDevKit_home/issues/141#issuecomment-252753181).

dalehenrich commented 7 years ago

... I thought that you were working on the client issue when you broke the master branch this time around ...

fniephaus commented 7 years ago

No, that I would've definitely done on the dev branch...

dalehenrich commented 7 years ago

I will need the client tests functional in a week or so, because I have work that depends upon that coming up... is there a way for me to use a version of SmalltalkCI that is properly functional?

dalehenrich commented 7 years ago

... if and when I need the client tests functional again ... I guess you have two months of work here?

fniephaus commented 7 years ago

I intend to fix this in the next few days (hopefully tomorrow). Do you really think it is that bad?

dalehenrich commented 7 years ago

I think that broken functionality on a master branch is bad ... this should never had been released until all of the tests were green and in the case of dumping stack traces for failed builds should have been reviewed before being released ...

dalehenrich commented 7 years ago

I did not schedule time for dealing with smalltalkCI right now and I've spent a lot of time on this over the last few days ... time that I really don't have ... and if I need to run client tests before you can figure out what's broken, what do you expect me to do?

fniephaus commented 7 years ago

Ok good, I think I will revert all changes tomorrow and move things back to dev until we found a better way to develop and tests smalltalkCI + GsDevKit_*. I really need to get some rest now...

dalehenrich commented 7 years ago

As I think about this ... the project that I'll need client tests for does not use Pharo3, so the broken Pharo3 client test is not as critical as I made it sound ...

The stack trace on failed tests is still very ugly and I really don't like the fact that this looks like a bug in GemStone, when it is really a case of premature release ... and I can't think of any solution that doesn't involve me having to do additional work ...

I'm sure that if you didn't have an exit code for Squeak you wouldn't have released the code to master until you had an acceptable solution ... so intimating that this problem is due to the lack of a "better way to develop and tests smalltalkCI + GsDevKit_*" is not correct at all.

If you hadn't been in such a hurry to release the code to master none of this mess over the last two days would have happened. You would have waited for me to come up with a solution and if it took time, because I don't have the time to figure out an acceptable solution then so be it ...

dalehenrich commented 7 years ago

I want to remind you of the environment variables that I added to SmalltalkCI to cover just this case ... I think it was your hurry and impatience to release this feature into production, not that we need a "better way to develop and tests smalltalkCI + GsDevKit_*"

Remember that by the time this came up you'd already pushed things to the master branch and were running around trying to patch your mistakes ...

dalehenrich commented 7 years ago

@fniephaus When you revert the smalltalkCI changes, I guess we need to revert the changes that removed the python formatter for GsDevKit_home as well, along with???

If you are interested in actually testing things before reverting (which I strongly recommend), and find places where the GsDevKit_home scripts need to be reverted (I've lost track of just what you actually ended up changing), I will help you set up your revert branch to point to a dev branch of GsDevKit_home so that you can get the tests passing before pushing the revert to the master branch and when we understand the changes that need to be made we can also determine the order of changes --- this is the procedure I've followed in the past when I've had to coordinate changes between dependent projects:

A similar trick can be arranged if needed for testing the revert of SmalltalkCI.

We should be able to do a similar trick with GsDevKit_home and test it with your revert branch before you push the revert branch to master --- also highly recommended.

GsDevKit / GsDevKit_home

smalltalkCI: JUnit XML Prettifier #141