search

ynput / ayon-houdini

Houdini addon for AYON
Apache License 2.0
11 stars 9 forks source link

Arnold renders from houdini is crashing with ayon 1.1.0 #100

Closed krishnaavril closed 1 month ago

krishnaavril commented 2 months ago

Is there an existing issue for this?

Current Behavior:

When submitting the arnold renders with ayon plugin 1.1.0 in deadline is giving us renders 61b3c34030dd0e2a55f16e6583c15ff98dc545c6

Expected Behavior:

Arnold renders should work with latest launcher 1.1.0

Version

1.0.0

What platform you are running on?

Windows

Steps To Reproduce:

Create arnold node in houdini Submit renders throw ayon to deadline render should get done without any errors

Are there any labels you wish to add?

Relevant log output:

Error
=======================================================
Error: Global job preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\avril7957\plugins\66e3f166b06103ee6a827c3c\GlobalJobPreLoad.py': RuntimeError : Failed to run AYON process to extract environments. (FranticX.Scripting.PythonNetException)
  File "C:\ProgramData\Thinkbox\Deadline10\workers\avril7957\plugins\66e3f166b06103ee6a827c3c\GlobalJobPreLoad.py", line 659, in __main__
    inject_ayon_environment(deadlinePlugin)
  File "C:\ProgramData\Thinkbox\Deadline10\workers\avril7957\plugins\66e3f166b06103ee6a827c3c\GlobalJobPreLoad.py", line 542, in inject_ayon_environment
    "Failed to run AYON process to extract environments."
   at Python.Runtime.PyObject.Invoke(PyObject[] args)
   at Python.Runtime.PyObject.InvokeMethod(String name, PyObject[] args)
   at FranticX.Scripting.PythonNetScriptEngine.CallFunction(String moduleName, String functionName, Object[] args)
   at Deadline.Plugins.PluginWrapper.d()
   at Deadline.Plugins.PluginWrapper.StartJob(String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgq, CancellationToken bgr)
   at Deadline.Plugins.SandboxedPlugin.StartJob(Job job, CancellationToken cancellationToken)
   at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajv, CancellationToken ajw)

=======================================================
Log
=======================================================
2024-09-13 13:36:15:  0: Loading Job's Plugin timeout is Disabled
2024-09-13 13:36:15:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'render'
2024-09-13 13:36:16:  0: Executing plugin command of type 'Initialize Plugin'
2024-09-13 13:36:16:  0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\avril7957\plugins\66e3f166b06103ee6a827c3c\Houdini.py'
2024-09-13 13:36:17:  0: INFO: Plugin execution sandbox using Python version 3
2024-09-13 13:36:17:  0: INFO: About: Houdini Plugin for Deadline
2024-09-13 13:36:17:  0: INFO: The job's environment will be merged with the current environment before rendering
2024-09-13 13:36:17:  0: Done executing plugin command of type 'Initialize Plugin'
2024-09-13 13:36:17:  0: Start Job timeout is disabled.
2024-09-13 13:36:17:  0: Task timeout is disabled.
2024-09-13 13:36:17:  0: Loaded job: Brz_SH_6210_fx_v019.hip - arnold_ropREN_FLY_L01_RE [RENDER] (66e3f166b06103ee6a827c3c)
2024-09-13 13:36:17:  0: Skipping I: because it is already mapped
2024-09-13 13:36:17:  0: Skipping J: because it is already mapped
2024-09-13 13:36:17:  0: Skipping K: because it is already mapped
2024-09-13 13:36:17:  0: Skipping H: because it is already mapped
2024-09-13 13:36:17:  0: Skipping P: because it is already mapped
2024-09-13 13:36:17:  0: Executing plugin command of type 'Start Job'
2024-09-13 13:36:17:  0: DEBUG: S3BackedCache Client is not installed.
2024-09-13 13:36:17:  0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\avril7957\plugins\66e3f166b06103ee6a827c3c\GlobalAssetTransferPreLoad.py'
2024-09-13 13:36:17:  0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2024-09-13 13:36:17:  0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2024-09-13 13:36:17:  0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2024-09-13 13:36:17:  0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2024-09-13 13:36:17:  0: INFO: Executing global job preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\avril7957\plugins\66e3f166b06103ee6a827c3c\GlobalJobPreLoad.py'
2024-09-13 13:36:17:  0: PYTHON: *** GlobalJobPreload 1.1.0 start ...
2024-09-13 13:36:17:  0: PYTHON: >>> Getting job ...
2024-09-13 13:36:17:  0: PYTHON: >>> Injecting AYON environments ...
2024-09-13 13:36:17:  0: PYTHON: --- Ayon executable: \\avril\softwares\applications\Ynput\AYON 1.0.4\ayon_console.exe
2024-09-13 13:36:17:  0: PYTHON: >>> Temporary path: C:\Users\RENDER~1.001\AppData\Local\Temp\20240913080617270422_0d93b898-71a7-11ef-8426-ac50de58f4a0.json
2024-09-13 13:36:17:  0: INFO: Setting Process Environment Variable AYON_SERVER_URL to http://192.168.168.168:5000
2024-09-13 13:36:17:  0: INFO: Setting Process Environment Variable AYON_API_KEY to 87866d2d8f70409282d853ec479db6c7
2024-09-13 13:36:17:  0: INFO: Setting Process Environment Variable AYON_BUNDLE_NAME to LBS_0087
2024-09-13 13:36:17:  0: PYTHON: >>> Executing: \\avril\softwares\applications\Ynput\AYON 1.0.4\ayon_console.exe --headless addon applications extractenvironments C:\Users\RENDER~1.001\AppData\Local\Temp\20240913080617270422_0d93b898-71a7-11ef-8426-ac50de58f4a0.json --envgroup farm --project Barroz --folder /RA_08/SH_6210 --task fx --app houdini/20.0.547
2024-09-13 13:36:22:  0: STDOUT: Python path configuration:
2024-09-13 13:36:22:  0: STDOUT:   PYTHONHOME = (not set)
2024-09-13 13:36:22:  0: STDOUT:   PYTHONPATH = (not set)
2024-09-13 13:36:22:  0: STDOUT:   program name = '\\avril\softwares\applications\Ynput\AYON 1.1.0\ayon_console.exe'
2024-09-13 13:36:22:  0: STDOUT:   isolated = 0
2024-09-13 13:36:22:  0: STDOUT:   environment = 0
2024-09-13 13:36:22:  0: STDOUT:   user site = 1
2024-09-13 13:36:22:  0: STDOUT:   import site = 0
2024-09-13 13:36:22:  0: STDOUT:   sys._base_executable = '\\\\avril\\softwares\\applications\\Ynput\\AYON 1.1.0\\ayon_console.exe'
2024-09-13 13:36:22:  0: STDOUT:   sys.base_prefix = ''
2024-09-13 13:36:22:  0: STDOUT:   sys.base_exec_prefix = ''
2024-09-13 13:36:22:  0: STDOUT:   sys.platlibdir = 'lib'
2024-09-13 13:36:22:  0: STDOUT:   sys.executable = '\\\\avril\\softwares\\applications\\Ynput\\AYON 1.1.0\\ayon_console.exe'
2024-09-13 13:36:22:  0: STDOUT:   sys.prefix = ''
2024-09-13 13:36:22:  0: STDOUT:   sys.exec_prefix = ''
2024-09-13 13:36:22:  0: STDOUT:   sys.path = [
2024-09-13 13:36:22:  0: STDOUT:     '\\\\avril\\softwares\\applications\\Ynput\\AYON 1.1.0\\lib\\library.zip',
2024-09-13 13:36:22:  0: STDOUT:     '\\\\avril\\softwares\\applications\\Ynput\\AYON 1.1.0\\lib',
2024-09-13 13:36:22:  0: STDOUT:   ]
2024-09-13 13:36:22:  0: STDOUT: Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
2024-09-13 13:36:22:  0: STDOUT: Python runtime state: core initialized
2024-09-13 13:36:22:  0: STDOUT: ModuleNotFoundError: No module named 'encodings'
2024-09-13 13:36:22:  0: STDOUT: Current thread 0x000036c4 (most recent call first):
2024-09-13 13:36:22:  0: STDOUT: <no Python frame>
2024-09-13 13:36:22:  0: STDOUT: >>> Connected to AYON server http://192.168.168.168:5000
2024-09-13 13:36:22:  0: PYTHON: Failed to run AYON process to extract environments. Trying to use legacy arguments.
2024-09-13 13:36:23:  0: STDOUT: Python path configuration:
2024-09-13 13:36:23:  0: STDOUT:   PYTHONHOME = (not set)
2024-09-13 13:36:23:  0: STDOUT:   PYTHONPATH = (not set)
2024-09-13 13:36:23:  0: STDOUT:   program name = '\\avril\softwares\applications\Ynput\AYON 1.1.0\ayon_console.exe'
2024-09-13 13:36:23:  0: STDOUT:   isolated = 0
2024-09-13 13:36:23:  0: STDOUT:   environment = 0
2024-09-13 13:36:23:  0: STDOUT:   user site = 1
2024-09-13 13:36:23:  0: STDOUT:   import site = 0
2024-09-13 13:36:23:  0: STDOUT:   sys._base_executable = '\\\\avril\\softwares\\applications\\Ynput\\AYON 1.1.0\\ayon_console.exe'
2024-09-13 13:36:23:  0: STDOUT:   sys.base_prefix = ''
2024-09-13 13:36:23:  0: STDOUT:   sys.base_exec_prefix = ''
2024-09-13 13:36:23:  0: STDOUT:   sys.platlibdir = 'lib'
2024-09-13 13:36:23:  0: STDOUT:   sys.executable = '\\\\avril\\softwares\\applications\\Ynput\\AYON 1.1.0\\ayon_console.exe'
2024-09-13 13:36:23:  0: STDOUT:   sys.prefix = ''
2024-09-13 13:36:23:  0: STDOUT:   sys.exec_prefix = ''
2024-09-13 13:36:23:  0: STDOUT:   sys.path = [
2024-09-13 13:36:23:  0: STDOUT:     '\\\\avril\\softwares\\applications\\Ynput\\AYON 1.1.0\\lib\\library.zip',
2024-09-13 13:36:23:  0: STDOUT:     '\\\\avril\\softwares\\applications\\Ynput\\AYON 1.1.0\\lib',
2024-09-13 13:36:23:  0: STDOUT:   ]
2024-09-13 13:36:23:  0: STDOUT: Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
2024-09-13 13:36:23:  0: STDOUT: Python runtime state: core initialized
2024-09-13 13:36:23:  0: STDOUT: ModuleNotFoundError: No module named 'encodings'
2024-09-13 13:36:23:  0: STDOUT: Current thread 0x0000a264 (most recent call first):
2024-09-13 13:36:23:  0: STDOUT: <no Python frame>
2024-09-13 13:36:23:  0: STDOUT: >>> Connected to AYON server http://192.168.168.168:5000
2024-09-13 13:36:23:  0: PYTHON: Traceback (most recent call last):
2024-09-13 13:36:23:  0: PYTHON:   File "C:\ProgramData\Thinkbox\Deadline10\workers\avril7957\plugins\66e3f166b06103ee6a827c3c\GlobalJobPreLoad.py", line 542, in inject_ayon_environment
2024-09-13 13:36:23:  0: PYTHON:     "Failed to run AYON process to extract environments."
2024-09-13 13:36:23:  0: PYTHON: RuntimeError: Failed to run AYON process to extract environments.
2024-09-13 13:36:23:  0: PYTHON: !!! Injection failed.
2024-09-13 13:36:25:  0: Done executing plugin command of type 'Start Job'

=======================================================
Details
=======================================================
Date: 09/13/2024 13:36:35
Frames: 1001-1005
Elapsed Time: 00:00:00:14
Job Submit Date: 09/13/2024 13:31:42
Job User: harinadh.m
Average RAM Usage: 42669846528 (42%)
Peak RAM Usage: 42669846528 (42%)
Average CPU Usage: 3%
Peak CPU Usage: 6%
Used CPU Clocks (x10^6 cycles): 43425
Total CPU Clocks (x10^6 cycles): 1447478

=======================================================
Worker Information
=======================================================
Worker Name: avril7957
Version: v10.2.1.1 Release (094cbe890)
Operating System: Windows 10 Pro
Running As Service: Yes
Machine User: render
IP Address: 192.168.0.26
MAC Address: 78:8C:B5:35:0C:5C
CPU Architecture: x64
CPUs: 32
CPU Usage: 10%
Memory Usage: 56.0 GB / 95.1 GB (58%)
Free Disk Space: 77.522 GB 
Video Card: NVIDIA GeForce RTX 3060 Ti

Additional context:

No response

krishnaavril commented 2 months ago

btw when we change the plugin to 1.0.4 in deadline ayon plugin, its rendering fine. I believe this is a ayon issue.

BigRoy commented 2 months ago

Please update your GlobalJobPreLoad and try again. You are on 1.1.0 but in deadline repo there's 1.1.3?

Also, on that worker - can you manually try the command line call like:

set AYON_SERVER_URL=http://192.168.168.168:5000
set AYON_API_KEY=87866d2d8f70409282d853ec479db6c7
set AYON_BUNDLE_NAME=LBS_0087

\\avril\softwares\applications\Ynput\AYON 1.0.4\ayon_console.exe --headless addon applications extractenvironments C:\Users\RENDER~1.001\AppData\Local\Temp\20240913080617270422_0d93b898-71a7-11ef-8426-ac50de58f4a0.json --envgroup farm --project Barroz --folder /RA_08/SH_6210 --task fx --app houdini/20.0.547

But then of course set other required environment variables if needed. And update that temp path to a valid location.

BigRoy commented 2 months ago

btw when we change the plugin to 1.0.4 in deadline ayon plugin, its rendering fine

Which plug-in, where? are you referring to GlobalJobPreLoad.py or?

krishnaavril commented 2 months ago

btw when we change the plugin to 1.0.4 in deadline ayon plugin, its rendering fine

Which plug-in, where? are you referring to GlobalJobPreLoad.py or?

image I was referring to this plugin,

Yes, I'll test with the latest GlobalJobPreload.py

antirotor commented 1 month ago

is this still an issue? If so, can you post how PATH looks like on farm?

krishnaavril commented 1 month ago

Sorry, I missed this. I'll test again and let you know.

we were using old globalpreload.py currently as the new one is giving some python print issues, I'll solve that and post the latest bug report here

krishnaavril commented 1 month ago

It's rendering fine now after updatingGlobalJobPreload.py

BigRoy commented 1 month ago

the new one is giving some python print issues, I'll solve that and post the latest bug report here

Could you elaborate on what you had to change? Are you running an old Deadline in Python 2 mode, or?

krishnaavril commented 1 month ago

Repository Version: 10.2.1.1 (094cbe890)

I think we are using python3. But I'm really not sure why it's giving error with print functions. basically comment the print function to get it to work, since its working I wasn't able to debug why it is causing the error, I'll have to debug that sometime when I get time...

Even old version of GlobalJobPreLoad.py we commented the print function

BigRoy commented 1 month ago

That Deadline version still supports Python 2 sandbox - so you MAY be running Python 2 there.

Can you check this setting: image

It should be set to 3.

We do have the "shebang" in the relevant files I believe as described here, for example here: https://github.com/ynput/ayon-deadline/blob/6f5b59fb6ebd02403fabcb859c48fd753cf3f2a3/client/ayon_deadline/repository/custom/plugins/GlobalJobPreLoad.py#L1

However, it seems that we may be missing an ! exclamation point there.

Our is:

ours:
# /usr/bin/env python3
in their example:
#!/usr/bin/env python3

So it might be worth it also to test whether adding that exclamation point there works for you.