search

aws-deadline / deadline-cloud-for-cinema-4d

AWS Deadline Cloud for Cinema 4D
Apache License 2.0
8 stars 9 forks source link

Bug: Unable to progress rendering #81

Open Waqiah opened 1 week ago

Waqiah commented 1 week ago

Expected Behaviour

To be able to set up a render farm using AWS Deadline Cloud's CMF with GPU instances.

Current Behaviour

Looking for assistance or recommended guides in setting up a render farm using GPU instances.

Initially had the following:

However, the rendering was not progressing. Next the software was re-installed and the driver installation was completed successfully following the instructions here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-nvidia-driver.html

However the issues persists. From the logs it was noted that there could be an issues with the permissions when calling ListFarms API or an issue with the job being submitted. It is understood that workers should be running the C4D adapter, but not the Deadline Client. The logs are from the worker seem to be running the adapter and using worker credentials which do not have access to the ListFarms API that that API availability check uses.

Next newly created and executed CMF worker on Windows Server 2022. The task file executed has had the OCIO-related settings removed, and it is a very small task file yet still not able to progress the rendering.

Snippets from the logs of the newly created worker: 2024/10/23 15시 37분 20초+09:00 ADAPTOR_OUTPUT: STDOUT: Performing action: {"name": "scene_file", "args": {"scene_file": "C:\ProgramData\Amazon\OpenJD\session-5cd7884013844df28118ca8fa5f7afd7hj2d8v2p\assetroot-e04c9fd3e29205d8cb7d\Untitled 2.c4d"}} ... 2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: ERROR:Could not activate 2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: res:5 2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: 2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: Error:18 2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: 2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: initOCIOconfig cfgFile: OCIO_ENV: 2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: 2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: Initialization error!!!! 2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: ... 2024/10/23 15시 37분 26초+09:00 ADAPTOR_OUTPUT: STDOUT: Error: application crashed ... ... and during the session cleanup ... 2024/10/23 15시 37분 28초+09:00 Process failed to start: [WinError 2] 지정된 파일을 찾을 수 없습니다.

There seems to be an issue with the scene file. Please let me know if you would like to view these logs. C4D is crashing due to the version of C4D that created the scene file isn't the same as the version that's running on the worker. Would resolving the issue with the scene file potentially resolve the WinError 2 (file not found) error as well?

Reproduction Steps

Please let me know what information is needed to replicate this issue.

Code Snippet

Please let me know which code or files are needed.

karthikbekalp commented 1 week ago

Hi @Waqiah , is it possible for you to provide more worker logs to understand this issue better?

Waqiah commented 1 week ago

Hello,

Sure, I'll attach the log file: logfile.txt

karthikbekalp commented 1 week ago

Thanks for adding your log file. I discussed this with @jblagden and it looks like the primary issue is due to the scene file: 

2024/10/23 15시 37분 20초+09:00 ADAPTOR_OUTPUT: STDOUT: Performing action: {"name": "scene_file", "args": {"scene_file": "C:\\ProgramData\\Amazon\\...\\session-abcdefg\\assetroot-e04c9fd3e29205d8cb7d\\Untitled 2.c4d"}}
2024/10/23 15시 37분 20초+09:00 ADAPTOR_OUTPUT: STDOUT: WARNING: Invalid shader cache item 0x000001C2A27CB480 [dpdx_shader_cache.cpp(452)]
2024/10/23 15시 37분 20초+09:00 ADAPTOR_OUTPUT: STDOUT: WARNING: Invalid shader cache item 0x000001C2A27CB9C0 [dpdx_shader_cache.cpp(452)]
2024/10/23 15시 37분 20초+09:00 ADAPTOR_OUTPUT: STDOUT: WARNING: Invalid shader cache item 0x000001C2A27CBB80 [dpdx_shader_cache.cpp(452)]
2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: ERROR:Could not activate
2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: res:5
2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: 
2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: Error:18
2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: 
2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: initOCIOconfig     cfgFile:    OCIO_ENV:
2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: 
2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: Initialization error!!!!
2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: 
2024/10/23 15시 37분 24초+09:00 ADAPTOR_OUTPUT: STDOUT: Performing action: {"name": "take", "args": {"take": "Main"}}

Does this happen to all scene files or just this one?

The final error is C4D is crashing after complaining that the version of C4D that created the scene file isn't the same as the version that's running on the worker:

2024/10/23 15시 37분 26초+09:00 ADAPTOR_OUTPUT: STDOUT: Scene created in plugin version 12000025
2024/10/23 15시 37분 26초+09:00 ADAPTOR_OUTPUT: STDOUT: 
2024/10/23 15시 37분 26초+09:00 ADAPTOR_OUTPUT: STDOUT: Scene plugin version is older than version:2023.1.3-[R5]
2024/10/23 15시 37분 26초+09:00 ADAPTOR_OUTPUT: STDOUT:  Plugin will convert the parameters of displacement texture.
2024/10/23 15시 37분 26초+09:00 ADAPTOR_OUTPUT: STDOUT:  Animated attributes should be changed by manually!
2024/10/23 15시 37분 26초+09:00 ADAPTOR_OUTPUT: STDOUT: 
2024/10/23 15시 37분 26초+09:00 ADAPTOR_OUTPUT: STDOUT: Error: application crashed
2024/10/23 15시 37분 26초+09:00 ADAPTOR_OUTPUT: STDOUT: C4DUnhandledExceptionFilter: writing exception info
2024/10/23 15시 37분 26초+09:00 ADAPTOR_OUTPUT: STDOUT: C4DUnhandledExceptionFilter: writing call stacks
2024/10/23 15시 37분 27초+09:00 test_fail: Error encountered while running adaptor: Cinema4D exited early and did not render successfully, please check render logs. Exit code 1
2024/10/23 15시 37분 27초+09:00 ERROR: Entrypoint failed: test_fail: Error encountered while running adaptor: Cinema4D exited early and did not render successfully, please check render logs. Exit code 1
2024/10/23 15시 37분 27초+09:00 Process pid 8756 exited with code: 1 (unsigned) / 0x1 (hex)
2024/10/23 15시 37분 27초+09:00  
2024/10/23 15시 37분 27초+09:00 ==============================================

And after this error the Worker fails to clean up the session:

2024/10/23 15시 37분 28초+09:00 ==============================================
2024/10/23 15시 37분 28초+09:00 --------- Session Cleanup
2024/10/23 15시 37분 28초+09:00 ==============================================
2024/10/23 15시 37분 28초+09:00 Deleting working directory: C:\ProgramData\Amazon\...\session-abcdefg
2024/10/23 15시 37분 28초+09:00 Running command start \"Powershell\" /high /wait /b powershell -Command Remove-Item -Recurse -Force "C:\ProgramData\Amazon\...\session-abcdefg\assetroot-e04c9fd3e29205d8cb7d, C:\ProgramData\Amazon\...\session-abcdefg\embedded_filesf3j4popv, C:\ProgramData\Amazon\...\session-5cd7884013844df28118ca8fa5f7afd7hj2d8v2p\tmpfqyyh38n.json, C:\ProgramData\Amazon\...\session-5cd7884013844df28118ca8fa5f7afd7hj2d8v2p\tmpkf4fhmch.json, C:\ProgramData\Amazon\...\session-5cd7884013844df28118ca8fa5f7afd7hj2d8v2p\tmpo8fe5t4e.json"
2024/10/23 15시 37분 28초+09:00 Process failed to start: [WinError 2]

WinError 2 is "The specified file could not be found.", but it's not clear which of the files in the command are missing.

All this points to an issue with the scene file if this is the only one that has issues, or an issue with the CMF setup.

I recommend making sure the version of C4D that created the scene file matches what's on the CMF worker. That'll resolve the second highlighted issue and possibly the first.