no - Githubissues

unittech commented 2 years ago

Hi there

we're tryiing to run this on CentOS 7.6 NVIDIA-SMI 460.91.03 RTX A6000 flame_2022.2

we get this error

Received 1 clip to process, press Ctrl+C to cancel Initializing duplicate frames interpolation... Trained model loaded: /home/flame/flameTimewarpML/bundle/trained_models/default/v2.4.model Total frames: 0%| | 0/750 [00:00<?, ?frame/s('CUDA error: no kernel image is available for execution on the device in '
"<class 'RuntimeError'>") ['Traceback (most recent call last):\n', ' File "/home/flame/flameTimewarpML/bundle/inference_dpframes.py", line 298, ' 'in \n' ' ICurrent = F.pad(ICurrent, padding)\n', ' File ' '"/home/flame/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", ' 'line 3553, in _pad\n' ' return _VF.constant_pad_nd(input, pad, value)\n', 'RuntimeError: CUDA error: no kernel image is available for execution on the ' 'device\n'] Traceback (most recent call last): File "/home/flame/flameTimewarpML/bundle/inference_dpframes.py", line 298, in ICurrent = F.pad(ICurrent, padding) File "/home/flame/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 3553, in _pad return _VF.constant_pad_nd(input, pad, value) RuntimeError: CUDA error: no kernel image is available for execution on the device Press Enter to continue...Exception ignored in thread started by: <function build_read_buffer at 0x7f4f54daf280> Traceback (most recent call last): File "/home/flame/flameTimewarpML/bundle/inference_dpframes.py", line 97, in build_read_buffer frame_data = cv2.imread(os.path.join(user_args.input, frame), cv2.IMREAD_COLOR | cv2.IMREAD_ANYDEPTH)[:, :, ::-1].copy() TypeError: 'NoneType' object is not subscriptable

and in the flame log it shows this

[flameTimewarpML] [flameAppFramework] waking up [flameTimewarpML] preferences loaded from /home/flame/.config/flameTimewarpML/ep01/flameTimewarpML.IB.xtfx1.prefs [flameTimewarpML] preferences loaded from /home/flame/.config/flameTimewarpML/ep01/flameTimewarpML.IB.prefs [flameTimewarpML] preferences loaded from /home/flame/.config/flameTimewarpML/ep01/flameTimewarpML.prefs [flameTimewarpML] checking existing bundle id /home/flame/flameTimewarpML/bundle/bundle_id [flameTimewarpML] env bundle already exists with id matching current version PYTHON : flameTimewarpML initializing [flameTimewarpML] creating folders: /var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A/source [flameTimewarpML] Executing command: echo "/var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A">/home/flame/flameTimewarpML/bundle/locks/B9EEB2FEBB2B44EF6D0D47EFAD7FDE6DB1AD8132.lock [flameTimewarpML] Executing command: konsole -e /bin/bash -c 'eval "$(/home/flame/flameTimewarpML/miniconda3/bin/conda shell.bash hook)"; conda activate; cd /home/flame/flameTimewarpML/bundle; echo "Received 1 clip to process, press Ctrl+C to cancel"; trap exit SIGINT SIGTERM; python3 /home/flame/flameTimewarpML/bundle/inference_dpframes.py --model /home/flame/flameTimewarpML/bundle/trained_models/default/v2.4.model --input /var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A/source --output /var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A; echo "Commands finished. You can close this window"' [flameTimewarpML] Importing result from: /var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A [flameTimewarpML] Cleaning up temporary files used: ['/var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A/source'] [flameTimewarpML] Executing command: rm -f "/var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A/source/" Exception in thread Thread-2: Traceback (most recent call last): File "/opt/Autodesk/python/2022.2/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/opt/Autodesk/python/2022.2/lib/python3.7/threading.py", line 870, in run self._target(self._args, **self._kwargs) File "/opt/Autodesk/shared/python/flameTimewarpML.py", line 3138, in raise_last_window if 'Flame' not in line: TypeError: a bytes-like object is required, not 'str'

talosh commented 2 years ago

Hi, you’d need to update pytorch for cuda11+ to be able to use it with Ampere GPUs.

cd /home//flametimewarpml/bundle ./init_env pip install torch==1.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html https://download.pytorch.org/whl/torch_stable.html

Let me know if it helps, Andriy.

On 5 Jan 2022, at 15:42, unittech @.***> wrote:

Hi there

we're tryiing to run this on CentOS 7.6 NVIDIA-SMI 460.91.03 RTX A6000 flame_2022.2

we get this error

Received 1 clip to process, press Ctrl+C to cancel Initializing duplicate frames interpolation... Trained model loaded: /home/flame/flameTimewarpML/bundle/trained_models/default/v2.4.model Total frames: 0%| | 0/750 [00:00<?, ?frame/s('CUDA error: no kernel image is available for execution on the device in ' "<class 'RuntimeError'>") ['Traceback (most recent call last):\n', ' File "/home/flame/flameTimewarpML/bundle/inference_dpframes.py", line 298, ' 'in \n' ' ICurrent = F.pad(ICurrent, padding)\n', ' File ' '"/home/flame/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", ' 'line 3553, in _pad\n' ' return _VF.constant_pad_nd(input, pad, value)\n', 'RuntimeError: CUDA error: no kernel image is available for execution on the ' 'device\n'] Traceback (most recent call last): File "/home/flame/flameTimewarpML/bundle/inference_dpframes.py", line 298, in ICurrent = F.pad(ICurrent, padding) File "/home/flame/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 3553, in _pad return _VF.constant_pad_nd(input, pad, value) RuntimeError: CUDA error: no kernel image is available for execution on the device Press Enter to continue...Exception ignored in thread started by: <function build_read_buffer at 0x7f4f54daf280> Traceback (most recent call last): File "/home/flame/flameTimewarpML/bundle/inference_dpframes.py", line 97, in build_read_buffer frame_data = cv2.imread(os.path.join(user_args.input, frame), cv2.IMREAD_COLOR | cv2.IMREAD_ANYDEPTH)[:, :, ::-1].copy() TypeError: 'NoneType' object is not subscriptable

and in the flame log it shows this

[flameTimewarpML] [flameAppFramework] waking up [flameTimewarpML] preferences loaded from /home/flame/.config/flameTimewarpML/ep01/flameTimewarpML.IB.xtfx1.prefs [flameTimewarpML] preferences loaded from /home/flame/.config/flameTimewarpML/ep01/flameTimewarpML.IB.prefs [flameTimewarpML] preferences loaded from /home/flame/.config/flameTimewarpML/ep01/flameTimewarpML.prefs [flameTimewarpML] checking existing bundle id /home/flame/flameTimewarpML/bundle/bundle_id [flameTimewarpML] env bundle already exists with id matching current version PYTHON : flameTimewarpML initializing [flameTimewarpML] creating folders: /var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A/source [flameTimewarpML] Executing command: echo "/var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A">/home/flame/flameTimewarpML/bundle/locks/B9EEB2FEBB2B44EF6D0D47EFAD7FDE6DB1AD8132.lock [flameTimewarpML] Executing command: konsole -e /bin/bash -c 'eval "$(/home/flame/flameTimewarpML/miniconda3/bin/conda shell.bash hook)"; conda activate; cd /home/flame/flameTimewarpML/bundle; echo "Received 1 clip to process, press Ctrl+C to cancel"; trap exit SIGINT SIGTERM; python3 /home/flame/flameTimewarpML/bundle/inference_dpframes.py --model /home/flame/flameTimewarpML/bundle/trained_models/default/v2.4.model --input /var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A/source --output /var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A; echo "Commands finished. You can close this window"' [flameTimewarpML] Importing result from: /var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A [flameTimewarpML] Cleaning up temporary files used: ['/var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A/source'] [flameTimewarpML] Executing command: rm -f "/var/tmp/SKIP_GAIA_GREECE_30secs_23Dec21_DUPFR_2022JAN05_1537_69A/source/" Exception in thread Thread-2: Traceback (most recent call last): File "/opt/Autodesk/python/2022.2/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/opt/Autodesk/python/2022.2/lib/python3.7/threading.py", line 870, in run self._target(self._args, **self._kwargs) File "/opt/Autodesk/shared/python/flameTimewarpML.py", line 3138, in raise_last_window if 'Flame' not in line: TypeError: a bytes-like object is required, not 'str'

— Reply to this email directly, view it on GitHub https://github.com/talosh/flameTimewarpML/issues/73, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMBGYKE5TY7XQKCWRQ3HOPDUURRHJANCNFSM5LKGXLMQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.

alatteri commented 2 years ago

Hi talosh,

We utilize a central install, with a mix of P , RTX and A architecture. Will updating to that version of pytorch break multi-generation compatability?

talosh commented 2 years ago

Hi Alan,

I think it is connected to driver / cuda version rather then to a card, so as long as cuda version is the same on all machines it should be fine regardless of an actual gpu model

On 5 Jan 2022, at 16:16, Alan Latteri @.***> wrote:

Hi talosh,

We utilize a central install, with a mix of P , RTX and A architecture. Will updating to that version of pytorch break multi-generation compatability?

— Reply to this email directly, view it on GitHub https://github.com/talosh/flameTimewarpML/issues/73#issuecomment-1005867528, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMBGYKFNEAPE2AXDQM4QK5DUURVERANCNFSM5LKGXLMQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.

talosh / flameTimewarpML

no #73