fix smoketester backwards-check to not parse stdout [LUCENE-10579]

[asfimport]

The smoketester parses the output of TestBackwardsCompatibility -verbose looking for certain prints for each index release.

But I think this is a noisier channel than you might expect. I added a hack to log the stuff its trying to parse... it is legit crazy. See attachment

Let's rethink, maybe we should just examine the zip files?

---

Migrated from LUCENE-10579 by Robert Muir (@rmuir), resolved May 20 2022 Attachments: backwards.log.gz Pull requests: https://github.com/apache/lucene/pull/903

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I attached compressed file of what the smoketester is parsing with regexps today. I guarantee it is wilder than you would imagine looking at the code.

I simply added this patch to log it:

   stdout = stdout.decode('utf-8',errors='replace').replace('\r\n','\n')
+  with open('%s/backwards.log' % unpackPath, 'w') as logfile:
+    logfile.write(stdout)

And now you can look at the 28.4MB of output that it parses.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

There's all kinds of stuff being printed, but this gives you an idea of what the 28.4MB looks like.

So I'm not surprised if this smoketester check fails here and there, its such a noisy channel. All it takes is something like MockRandomMergePolicy or some other component logging from another thread to prevent that multiline regexp from doing the right thing?

ESC[2AESC[1m<ESC[0;32;1m===========ESC[0;39;1m--> 92% EXECUTING [26s]ESC[mESC[35DESC[1BESC[1m> :lucene:backward-codecs:testESC[mESC[30DESC[1BESC[2AESC[1m<ESC[0;32;1m===========ESC[0;39;1m--> 92% EXECUTING [27s]ESESC[mESC[35DESC[2BESC[1AESC[1m> :lucene:backward-codecs:test > 0 tests completedESC[mESC[50DESC[1B
ESC[3AESC[35CESC[0KESC[35DESC[2BESC[1m> :lucene:backward-codecs:test > Executing test org.apache.lucene.backward_indeESC[mESC[79DESC[1BESC[3AESC[1m<ESC[0;32;1m===========ESC[0;39;1m--> 92% EXECUTING [28s]ESC[mESC[35DESC[3BESC[3AESC[0K
ESC[1m> Task :lucene:backward-codecs:testESC[mESC[0K
  1> filesystem: ExtrasFS(HandleLimitFS(LeakFS(ShuffleFS(DisableFsyncFS(VerboseFS(sun.nio.fs.LinuxFileSystemProvider@7764d0d3))))))ESC[0K
  1> FS 0 [2022-05-18T19:37:29.645632Z; SUITE-TestBackwardsCompatibility-seed#[2EBBD700BDB7349D]-worker]: createDirectory: ../../../../../../../../lucene_gradle (FAILED: java.nio.file.FileAlreadyExistsException: /tmp/lucene_gradle)
  1> Loaded codecs: [Lucene92, Asserting, CheapBastard, DeflateWithPresetCompressingStoredFieldsData, FastCompressingStoredFieldsData, FastDecompressionCompressingStoredFieldsData, HighCompressionCompressingStoredFieldsData, LZ4WithPresetCompressingStoredFieldsData, DummyCompressingStoredFieldsData, SimpleText, Lucene80, Lucene84, Lucene86, Lucene87, Lucene70, Lucene90, Lucene91]
  1> Loaded postingsFormats: [Lucene90, MockRandom, RAMOnly, LuceneFixedGap, LuceneVarGapFixedInterval, LuceneVarGapDocFreqInterval, TestBloomFilteredLucenePostings, Asserting, UniformSplitRot13, STUniformSplitRot13, BlockTreeOrds, BloomFilter, Direct, FST50, UniformSplit, SharedTermsUniformSplit, Lucene50, Lucene84]
  1> FS 0 [2022-05-18T19:37:29.780830Z; SUITE-TestBackwardsCompatibility-seed#[2EBBD700BDB7349D]-worker]: createDirectory: ../../../../../../../../lucene_gradle/lucene.backward_index.TestBackwardsCompatibility_2EBBD700BDB7349D-001
  1> FS 0 [2022-05-18T19:37:29.783274Z; SUITE-TestBackwardsCompatibility-seed#[2EBBD700BDB7349D]-worker]: createDirectory: ../../../../../../../../lucene_gradle/lucene.backward_index.TestBackwardsCompatibility_2EBBD700BDB7349D-001/8.0.0-cfs-001
  1> FS 0 [2022-05-18T19:37:29.785704Z; SUITE-TestBackwardsCompatibility-seed#[2EBBD700BDB7349D]-worker]: createDirectory: ../../../../../../../../lucene_gradle/lucene.backward_index.TestBackwardsCompatibility_2EBBD700BDB7349D-001/8.0.0-cfs-001 (FAILED: java.nio.file.FileAlreadyExistsException: /tmp/lucene_gradle/lucene.backward_index.TestBackwardsCompatibility_2EBBD700BDB7349D-001/8.0.0-cfs-001)
  1> FS 0 [2022-05-18T19:37:29.789291Z; SUITE-TestBackwardsCompatibility-seed#[2EBBD700BDB7349D]-worker]: newOutputStream[]: ../../../../../../../../lucene_gradle/lucene.backward_index.TestBackwardsCompatibility_2EBBD700BDB7349D-001/8.0.0-cfs-001/_0.cfe

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

or even maybe a gradle status update with its escape characters and so on (it has a progress bar and such), seems like that could be enough to break the check.

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 2090ac431843f689ca6f124723ef37c3b12a2826 in lucene's branch refs/heads/main from Robert Muir https://gitbox.apache.org/repos/asf?p=lucene.git;h=2090ac43184

LUCENE-10579: fix smoketester backwards-check to not parse stdout (#903)

This is very noisy, can contain gradle status updates, various other tests.verbose prints from other threads, you name it.

It causes the check to be flaky, and randomly "miss" seeing a test that executed.

Instead, let's look at the zip files. We can still preserve the essence of what the test wants to do, but without any flakiness.

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 5a5c8bca4e691d91801d04d881aa25560413835c in lucene's branch refs/heads/branch_9x from Robert Muir https://gitbox.apache.org/repos/asf?p=lucene.git;h=5a5c8bca4e6

LUCENE-10579: fix smoketester backwards-check to not parse stdout (#903)

This is very noisy, can contain gradle status updates, various other tests.verbose prints from other threads, you name it.

It causes the check to be flaky, and randomly "miss" seeing a test that executed.

Instead, let's look at the zip files. We can still preserve the essence of what the test wants to do, but without any flakiness.

[asfimport]

Ignacio Vera (@iverase) (migrated from JIRA)

Closing after the 9.3.0 release