search

launchableinc / cli

Launchable CLI
https://pypi.org/project/launchable/
Apache License 2.0
24 stars 13 forks source link

Improve the warning of CLI #786

Open ono-max opened 9 months ago

ono-max commented 9 months ago

https://github.com/ruby/ruby/actions/runs/7885458586/job/21516643427?pr=9777#step:10:83

When executing the launchable record build command, we don't know what happens. We need to improve it. 

WARNING: Failed to process a change to a file
java.io.UncheckedIOException: org.eclipse.jgit.errors.MissingObjectException: Missing blob c7f57385b32005caa5512d52f72135a5dc1df4de
    at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:38)
    at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transform(CommitGraphCollector.java:410)
    at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transfer(CommitGraphCollector.java:343)
    at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:195)
    at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:129)
    at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:149)
    at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:77)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob c7f57385b32005caa5512d52f72135a5dc1df4de
    at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
    at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:112)
    at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:257)
    at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1074)
    at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:1001)
    at org.eclipse.jgit.diff.DiffFormatter.format(DiffFormatter.java:704)
    at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:36)
    ... 6 more

Feb 13, 2024 11:05:02 AM com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository transform
WARNING: Failed to process a change to a file
java.io.UncheckedIOException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 64332a8f53ac12e252641da2dac966e5c5344e1e
    at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:38)
    at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transform(CommitGraphCollector.java:410)
    at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transfer(CommitGraphCollector.java:343)
    at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:195)
    at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:129)
    at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:149)
    at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:77)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob 64332a8f53ac12e252641da2dac966e5c5344e1e
    at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
    at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:112)
    at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:257)
    at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1074)
    at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:1001)
    at org.eclipse.jgit.diff.DiffFormatter.format(DiffFormatter.java:704)
    at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:36)
    ... 6 more

Feb 13, 2024 11:05:02 AM com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository transform
WARNING: Failed to process a change to a file
java.io.UncheckedIOException: org.eclipse.jgit.errors.MissingObjectException: Missing blob f2244ec20aa38532faa50243abe821486ca07244
    at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:38)
    at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transform(CommitGraphCollector.java:410)
    at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transfer(CommitGraphCollector.java:343)
    at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:195)
    at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:129)
    at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:149)
    at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:77)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob f2244ec20aa38532faa50243abe821486ca07244
    at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
    at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:112)
    at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:257)
    at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1074)
    at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:1001)
    at org.eclipse.jgit.diff.DiffFormatter.format(DiffFormatter.java:704)
    at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:36)
    ... 6 more

Feb 13, 2024 11:05:02 AM com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository transform
WARNING: Failed to process a change to a file
java.io.UncheckedIOException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 57306673cd3106a3fcbf204aa131dfced421bb1c
    at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:38)
    at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transform(CommitGraphCollector.java:410)
    at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transfer(CommitGraphCollector.java:343)
    at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:195)
    at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:129)
    at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:149)
    at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:77)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob 57306673cd3106a3fcbf204aa131dfced421bb1c
    at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
    at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:112)
    at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:257)
    at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1074)
    at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:1001)
    at org.eclipse.jgit.diff.DiffFormatter.format(DiffFormatter.java:704)
    at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:36)
    ... 6 more

Feb 13, 2024 11:05:02 AM com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository transform
WARNING: Failed to process a change to a file
java.io.UncheckedIOException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 20630e58097[83](https://github.com/ruby/ruby/actions/runs/7885458586/job/21516643427?pr=9777#step:10:84)c5e43e42c436e80e245e6bdb518
    at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:38)
    at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transform(CommitGraphCollector.java:410)
    at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transfer(CommitGraphCollector.java:343)
    at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:195)
    at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:129)
    at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:149)
    at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:77)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob 20630e5809783c5e43e42c436e80e245e6bdb518
    at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
    at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:112)
    at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:257)
    at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1074)
    at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:1001)
    at org.eclipse.jgit.diff.DiffFormatter.format(DiffFormatter.java:704)
    at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:36)
    ... 6 more

Launchable recorded 1 commit from repository /Users/runner/work/ruby/ruby/build
Launchable recorded build 16f8f23f114b11[85](https://github.com/ruby/ruby/actions/runs/7885458586/job/21516643427?pr=9777#step:10:86)70347e6b82e1b980c8a2ab7c to workspace ruby/ruby with commits from 1 repository:

| Name   | Path   | HEAD Commit                              |
|--------|--------|------------------------------------------|
| .      | .      | 60cc647c1983035c7998d1a6fbfd1c832aea3e[96](https://github.com/ruby/ruby/actions/runs/7885458586/job/21516643427?pr=9777#step:10:97) |

Visit https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/builds/2505254 to view this build and its test sessions

YAML file

https://github.com/ruby/ruby/blob/fc17ef891a3fcb12d4bf26394e093a021afe7658/.github/workflows/macos.yml

fetch-depth: 0 is configured so that Github does not use shallow clone.

ono-max commented 9 months ago

@kohsuke

Any ideas? I'm not familiar with this part.

kohsuke commented 9 months ago

From the log, I see that git fetch was run with --filter=blob:none According to https://stackoverflow.com/questions/49917616/what-is-the-git-clone-filter-options-syntax what this means is that BLOBs are not retrieved during this git-fetch. Instead, later git-clone appear to retrieve the needed objects. See https://git-scm.com/docs/partial-clone/2.29.0 for the overview of how this works internally in git.

That explains why BLOBs are reported as missing. The next question then is what can we do to retrieve those missing BLOBs as we run into them.

I checked JGit source repo and as of 6.8.0, there's no built-in support for this. So if we are to solve this, I think we'll walk the graph, identify missing objects, and run one bulk git-fetch.

kohsuke commented 9 months ago

Let's also look for workaround.