search

juno-fx / report

Ticket and reports repository
0 stars 0 forks source link

(Bug: Kuiper) X_Denoise failing on the farm #279

Open lamepennies opened 2 weeks ago

lamepennies commented 2 weeks ago

Dupe Check

Serivce this pertains to.

Kuiper

URL in Hubble.

https://ath.junovfx.com/workstations/

Browser

Chrome Based

Describe the bug

Hello, X_Denoise fails on the farm

======================================================= Error

Error: FailRenderException : [17:42.43] ERROR: X_Denoise.Merge4.disable: input already correct at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException) File "/var/lib/Thinkbox/Deadline10/workers/chester-0/plugins/6703041b64cbcf005aafa7d4/Nuke.py", line 264, in RenderTasks self.WritePython(scriptContents) File "/var/lib/Thinkbox/Deadline10/workers/chester-0/plugins/6703041b64cbcf005aafa7d4/Nuke.py", line 41, in WritePython self.WaitForProcess() File "/var/lib/Thinkbox/Deadline10/workers/chester-0/plugins/6703041b64cbcf005aafa7d4/Nuke.py", line 48, in WaitForProcess self.FlushMonitoredManagedProcessStdout( self.ProcessName ) File "/var/lib/Thinkbox/Deadline10/workers/chester-0/plugins/6703041b64cbcf005aafa7d4/Nuke.py", line 623, in HandleError self.deadlinePlugin.FailRender( self.GetRegexMatch( 0 ) ) at Python.Runtime.Dispatcher.Dispatch(ArrayList args) at __FranticX_GenericDelegate0Dispatcher.Invoke() at Deadline.Plugins.DeadlinePlugin.RenderTasks() at Deadline.Plugins.DeadlinePlugin.DoRenderTasks() at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel) at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

======================================================= Type

RenderPluginException

======================================================= Stack Trace

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgm, CancellationToken bgn) at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken) at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajt, CancellationToken aju)

======================================================= Log

2024-10-06 17:42:18: 0: Loading Job's Plugin timeout is Disabled 2024-10-06 17:42:19: 0: SandboxedPlugin: Render Job As User disabled, running as current user 'root' 2024-10-06 17:42:21: 0: Executing plugin command of type 'Initialize Plugin' 2024-10-06 17:42:21: 0: INFO: Executing plugin script '/var/lib/Thinkbox/Deadline10/workers/chester-0/plugins/6703041b64cbcf005aafa7d4/Nuke.py' 2024-10-06 17:42:21: 0: INFO: Plugin execution sandbox using Python version 3 2024-10-06 17:42:21: 0: INFO: About: Nuke Plugin for Deadline 2024-10-06 17:42:21: 0: INFO: The job's environment will be merged with the current environment before rendering 2024-10-06 17:42:21: 0: Done executing plugin command of type 'Initialize Plugin' 2024-10-06 17:42:21: 0: Start Job timeout is disabled. 2024-10-06 17:42:21: 0: Task timeout is disabled. 2024-10-06 17:42:21: 0: Loaded job: 001_03_Comp_3_v002.nk (6703041b64cbcf005aafa7d4) 2024-10-06 17:42:21: 0: Executing plugin command of type 'Start Job' 2024-10-06 17:42:21: 0: DEBUG: S3BackedCache Client is not installed. 2024-10-06 17:42:21: 0: INFO: Executing global asset transfer preload script '/var/lib/Thinkbox/Deadline10/workers/chester-0/plugins/6703041b64cbcf005aafa7d4/GlobalAssetTransferPreLoad.py' 2024-10-06 17:42:21: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer... 2024-10-06 17:42:21: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in /opt/Thinkbox/S3BackedCache/bin/task.py... 2024-10-06 17:42:21: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer. 2024-10-06 17:42:21: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system. 2024-10-06 17:42:21: 0: INFO: Scrubbing the LD and DYLD LIBRARY paths 2024-10-06 17:42:21: 0: INFO: Prepping OFX cache 2024-10-06 17:42:21: 0: INFO: Checking Nuke temp path: /var/tmp/nuke-u0 2024-10-06 17:42:21: 0: INFO: Path does not exist, creating it... 2024-10-06 17:42:21: 0: INFO: Path now exists 2024-10-06 17:42:21: 0: INFO: OFX cache prepped 2024-10-06 17:42:21: 0: INFO: Starting monitored managed process Nuke 2024-10-06 17:42:21: 0: INFO: Enable Path Mapping: True 2024-10-06 17:42:21: 0: INFO: Stdout Redirection Enabled: True 2024-10-06 17:42:21: 0: INFO: Asynchronous Stdout Enabled: False 2024-10-06 17:42:21: 0: INFO: Stdout Handling Enabled: True 2024-10-06 17:42:21: 0: INFO: Popup Handling Enabled: True 2024-10-06 17:42:21: 0: INFO: QT Popup Handling Enabled: False 2024-10-06 17:42:21: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False 2024-10-06 17:42:21: 0: INFO: Using Process Tree: True 2024-10-06 17:42:21: 0: INFO: Hiding DOS Window: True 2024-10-06 17:42:21: 0: INFO: Creating New Console: False 2024-10-06 17:42:21: 0: INFO: Running as user: root 2024-10-06 17:42:22: 0: INFO: Executable: "/apps/nuke/Nuke15.1v1/Nuke15.1" 2024-10-06 17:42:22: 0: INFO: Setting Process Environment Variable EDDY_DEVICE_LIST to 2024-10-06 17:42:22: 0: INFO: Argument: -V 2 -t "/var/lib/Thinkbox/Deadline10/workers/chester-0/jobsData/6703041b64cbcf005aafa7d4/thread0_tempEdeQG0/001_03_Comp_3_v002.nk" 2024-10-06 17:42:22: 0: INFO: Full Command: "/apps/nuke/Nuke15.1v1/Nuke15.1" -V 2 -t "/var/lib/Thinkbox/Deadline10/workers/chester-0/jobsData/6703041b64cbcf005aafa7d4/thread0_tempEdeQG0/001_03_Comp_3_v002.nk" 2024-10-06 17:42:22: 0: INFO: Startup Directory: "/apps/nuke/Nuke15.1v1" 2024-10-06 17:42:22: 0: INFO: Process Priority: BelowNormal 2024-10-06 17:42:22: 0: INFO: Process Affinity: default 2024-10-06 17:42:22: 0: INFO: Process is now running 2024-10-06 17:42:22: 0: Done executing plugin command of type 'Start Job' 2024-10-06 17:42:22: 0: Plugin rendering frame(s): 51-59 2024-10-06 17:42:22: 0: Executing plugin command of type 'Render Task' 2024-10-06 17:42:22: 0: INFO: Rendering write node Write2 2024-10-06 17:42:27: 0: STDOUT: Nuke 15.1v1, 64 bit, built Jun 6 2024. 2024-10-06 17:42:27: 0: STDOUT: Copyright (c) 2024 The Foundry Visionmongers Ltd. All Rights Reserved. 2024-10-06 17:42:30: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/_pathsetup.py 2024-10-06 17:42:30: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/init.tcl 2024-10-06 17:42:30: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/init.py 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/setenv.tcl 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/formats.tcl 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/caravr/init.py 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/getenv.tcl 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/OCIOColorSpace.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/exrReader.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Grade.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/jpgReader.tcl 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/jpegReader.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Reformat.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Transform.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Merge2.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/tifReader.tcl 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/tiffReader.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Mirror2.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Keymix.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Defocus.so 2024-10-06 17:42:31: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Tracker4.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Copy.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Premult.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/VectorGenerator.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/FrameBlend.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Clamp.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Median.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/IDistort.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Dissolve.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Colorspace.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Multiply.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Invert.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/ChannelMerge.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/TimeWarp.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/CurveTool.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Sampler.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/ColorLookup.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Remove.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Noise.so 2024-10-06 17:42:32: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/BlinkScript.so 2024-10-06 17:42:36: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Difference.so 2024-10-06 17:42:36: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/STMap.so 2024-10-06 17:42:36: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/TimeBlur.so 2024-10-06 17:42:36: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Assert.so 2024-10-06 17:42:36: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/CopyMetaData.so 2024-10-06 17:42:36: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/Crop.so 2024-10-06 17:42:36: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/OCIODisplay.so 2024-10-06 17:42:36: 0: STDOUT: Loading /apps/nuke/Nuke15.1v1/plugins/exrWriter.so 2024-10-06 17:42:36: 0: STDOUT: [17:42:36 EDT] Read nuke script: /var/lib/Thinkbox/Deadline10/workers/chester-0/jobsData/6703041b64cbcf005aafa7d4/thread0_tempEdeQG0/001_03_Comp_3_v002.nk 2024-10-06 17:42:36: 0: STDOUT: >>> [17:42.36] Warning: RotoPaint1: [PaintEngine] Abort/cancel checked with no valid trees. This op most likely needs to have its parent set correctly. 2024-10-06 17:42:36: 0: STDOUT: Writing /shows/ATH/66fb5fe4dab20b1067be20b2/v002/artifacts/Frames/Precomp/precompDenoise/4096x1716_exr/001_03_Comp_3_precompDenoise_v002.0051.exr 2024-10-06 17:42:36: 0: STDOUT: RotoPaint1 2024-10-06 17:42:37: 0: STDOUT: RotoPaint1 2024-10-06 17:42:38: 0: STDOUT: RotoPaint1 2024-10-06 17:42:39: 0: STDOUT: RotoPaint1 2024-10-06 17:42:43: 0: STDOUT: RotoPaint1 .1.3.5.7.9 2024-10-06 17:42:43: 0: STDOUT: Writing /shows/ATH/66fb5fe4dab20b1067be20b2/v002/artifacts/Frames/Precomp/precompDenoise/4096x1716_exr/001_03_Comp_3_precompDenoise_v002.0051.exr took 6.87 seconds 2024-10-06 17:42:43: 0: Done executing plugin command of type 'Render Task'

======================================================= Details

Date: 10/06/2024 17:42:53 Frames: 51-59 Elapsed Time: 00:00:00:35 Job Submit Date: 10/06/2024 17:41:46 Job User: tcomo Average RAM Usage: 2077761536 (4%) Peak RAM Usage: 6933196800 (11%) Average CPU Usage: 18% Peak CPU Usage: 80% Used CPU Clocks (x10^6 cycles): 224919 Total CPU Clocks (x10^6 cycles): 1249547

======================================================= Worker Information

Worker Name: chester-0 Version: v10.1.23.6 Release (773a6289d) Operating System: Ubuntu 22.04.4 LTS Machine User: root IP Address: 192.168.79.194 MAC Address: DE:38:E7:49:36:70 CPU Architecture: x86_64 CPUs: 16 CPU Usage: 100% Memory Usage: 6.5 GB / 61.7 GB (10%) Free Disk Space: 54.602 GB Video Card: Amazon.com, Inc. Device 1111

======================================================= AWS Information

Instance ID: i-06c11a0f76e94a98b Instance Type: m5a.4xlarge Image ID: ami-0e82877806425fbb1 Region: ap-south-1 Architecture: x86_64 Availability Zone: ap-south-1a

To reproduce

Tried to use X_Denoise to reduce noise the shot to grain and it's not rendering with the node in. Will try another node.

Expected behavior

No response

Screenshots

No response

Regression

No, this bug or issue has existed throughout my experience using Orion

Recent working Orion date

No response

Does this block you from using Orion daily?

No

lamepennies commented 2 weeks ago

This is part of the Compositors Toolkit BTW.

skeleturtle commented 2 weeks ago

@lamepennies

Tried taking a look into this a little bit what did run into some troubles. unfortunately the scene file that gets sent to the farm, gets overwritten once another version is sent. So first I will look to update our temp farm script file naming/location to try and avoid this to help with debugging in the future.

It seems like there are expressions on the x_denoise merge nodes which at initial glance don't seem to make much sense to be honest. As far as I can tell the merge nodes expression would always set it to be disabled. I started my own file, imported your scene and created a new x_denoise I was going to test some adjustments to the expression but It doing so caused nuke to crash both locally and on the farm when attempting to render. At this point I'd just guess I don't know how to properly set up the the node. Although the node does seem like it would work best with a GPU which our workstations also don't have.

I would like to take another look into the gizmo and potentially whats going on. If you get a chance and want to create a lighter weight test script with the node set up "properly" I can play around with that and try to pin point whats going on. In shot 03 you'll see a Light - Test task. Otherwise I'll try to review this tomorrow when I can.

lamepennies commented 2 weeks ago

Saved as v02. I didn't click any settings, just the default values.