Trying to trace function "__execute_sample_during_training" from https://github.com/Nerogar/OneTrainer
There are some errors at the end about \<string> and \<frozen importlib._bootstrap>:
execute_sample_during_training - put_events: so far: 918571
finalizing native tracer
queue length 0
put_file_slice
tracing_did_complete
got evt EventsSlice
queue length 1
got evt FileContentSlice
error! at store_file_contents:
[Errno 22] Invalid argument: ''
error! at store_file_contents:
[Errno 22] Invalid argument: ''
skip_to_free_header_chunk
pos = 12
after pos = 28
queue length 0
got evt StopCommand
skip_to_free_header_chunk
pos = 12
after pos = 52
metadata saved to C:\Users\Max\Desktop\OneTrainer\pycrunch-recordings\execute_sample_during_training\pycrunch-trace.meta.json
C:\Users\Max\Desktop\OneTrainer\venv\lib\site-packages\torch\autograd\graph.py:744: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
step: 100%|████████████████████████████████████████████████| 1/1 [09:36<00:00, 576.06s/it, loss=1.49, smooth loss=1.49]
epoch: 100%|████████████████████████████████████████████████████████████████████████████| 1/1 [09:38<00:00, 578.23s/it]
But files pycrunch-trace.meta.json and session.chunked.pycrunch-trace appear as expected. When I open them in app.pytrace.com, I see nothing at screen but error in console:
google-protobuf.js:
{
"message": "Failure: Invalid wire type: 6 (at position 6)",
"reportErrorToServer": true,
"messagePattern": "Failure: Invalid wire type: %s (at position %s)"
}
sorry, it seems these errors are not related to issues with protobuf. If I interrupt traced program execution early with a KeyboardInterrupt, the file opens without issues
Trying to trace function "__execute_sample_during_training" from https://github.com/Nerogar/OneTrainer There are some errors at the end about \<string> and \<frozen importlib._bootstrap>:
execute_sample_during_training - put_events: so far: 918571 finalizing native tracer queue length 0 put_file_slice tracing_did_complete got evt EventsSlice queue length 1 got evt FileContentSlice error! at store_file_contents:
[Errno 22] Invalid argument: ''
error! at store_file_contents:
[Errno 22] Invalid argument: ''
skip_to_free_header_chunk
pos = 12
after pos = 28
queue length 0
got evt StopCommand
skip_to_free_header_chunk
pos = 12
after pos = 52
metadata saved to C:\Users\Max\Desktop\OneTrainer\pycrunch-recordings\ execute_sample_during_training\pycrunch-trace.meta.json
C:\Users\Max\Desktop\OneTrainer\venv\lib\site-packages\torch\autograd\graph.py:744: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
step: 100%|████████████████████████████████████████████████| 1/1 [09:36<00:00, 576.06s/it, loss=1.49, smooth loss=1.49]
epoch: 100%|████████████████████████████████████████████████████████████████████████████| 1/1 [09:38<00:00, 578.23s/it]
But files pycrunch-trace.meta.json and session.chunked.pycrunch-trace appear as expected. When I open them in app.pytrace.com, I see nothing at screen but error in console: google-protobuf.js: { "message": "Failure: Invalid wire type: 6 (at position 6)", "reportErrorToServer": true, "messagePattern": "Failure: Invalid wire type: %s (at position %s)" }
pycrunch-trace v0.3 python 3.10
Something gone wrong?