search

o3de / o3de-extras

Other
58 stars 60 forks source link

Bug Report: Random Jenkins Build Failures at asset_profile step. #672

Open galibzon opened 4 months ago

galibzon commented 4 months ago

Describe the bug The o3de-extras jenkins jobs fail randomly during the asset_profile step. The failure is always the same, but it occurs randomly:

[2024-02-13T16:39:20.258Z] AssetProcessor: -----------------------------------------
[2024-02-13T16:39:20.258Z] AssetProcessor: Asset Processor Batch Processing complete
[2024-02-13T16:39:20.258Z] AssetProcessor: ---------------FAILED ASSETS-------------
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_0_depthpasstransparentmin.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_1_shadowmappass_customz.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_2_depthpass.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_3_shadowmappass.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_4_depthpasstransparentmax.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_5_depthpass_customz.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: -----------------------------------------
[2024-02-13T16:39:20.258Z] AssetProcessor: Number of Assets Successfully Processed: 2102.
[2024-02-13T16:39:20.258Z] AssetProcessor: Number of Assets Failed to Process: 6.

Some examples of jenkins jobs that have failed like that: https://jenkins.build.o3de.org/blue/organizations/jenkins/o3de-extras/detail/development/111/pipeline https://jenkins.build.o3de.org/blue/organizations/jenkins/o3de-extras/detail/PR-664/8/pipeline

Assets required N/A

Steps to reproduce Run a jenkins a job under the o3de-extras repo.

Expected behavior asset_profile should pass

Actual behavior asset_profile fails randomly, sometimes requiring to re-run the job at least 3 times for around 9hrs+ of waste compute time.

Screenshots/Video N/A

Found in Branch latest development of o3de-extras + o3de

Desktop/Device (please complete the following information):

Additional context Add any other context about the problem here.

amzn-changml commented 2 months ago

Note that this appears to pass if using a clean build: https://jenkins.build.o3de.org/blue/organizations/jenkins/o3de-extras/detail/development/123/pipeline, however the subsequent incremental runs fails again.

We may want to consider having Jenkins set this error as a non-blocking, rather than failing the AR run completely. Here's an example of how that's implemented in Linux: https://github.com/o3de/o3de/pull/8832