gleb-sevruk / pycrunch-trace

Time Travel Debugging for Python
https://pytrace.com
MIT License
152 stars 12 forks source link

App shows nothing #14

Open seniorsolt opened 1 month ago

seniorsolt commented 1 month ago

Trying to trace function "__execute_sample_during_training" from https://github.com/Nerogar/OneTrainer There are some errors at the end about \<string> and \<frozen importlib._bootstrap>:

execute_sample_during_training - put_events: so far: 918571 finalizing native tracer queue length 0 put_file_slice tracing_did_complete got evt EventsSlice queue length 1 got evt FileContentSlice error! at store_file_contents: [Errno 22] Invalid argument: '' error! at store_file_contents: [Errno 22] Invalid argument: '' skip_to_free_header_chunk pos = 12 after pos = 28 queue length 0 got evt StopCommand skip_to_free_header_chunk pos = 12 after pos = 52 metadata saved to C:\Users\Max\Desktop\OneTrainer\pycrunch-recordings\execute_sample_during_training\pycrunch-trace.meta.json C:\Users\Max\Desktop\OneTrainer\venv\lib\site-packages\torch\autograd\graph.py:744: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.) return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass step: 100%|████████████████████████████████████████████████| 1/1 [09:36<00:00, 576.06s/it, loss=1.49, smooth loss=1.49] epoch: 100%|████████████████████████████████████████████████████████████████████████████| 1/1 [09:38<00:00, 578.23s/it]

But files pycrunch-trace.meta.json and session.chunked.pycrunch-trace appear as expected. When I open them in app.pytrace.com, I see nothing at screen but error in console: google-protobuf.js: { "message": "Failure: Invalid wire type: 6 (at position 6)", "reportErrorToServer": true, "messagePattern": "Failure: Invalid wire type: %s (at position %s)" }

pycrunch-trace v0.3 python 3.10

Something gone wrong?

seniorsolt commented 1 month ago

sorry, it seems these errors are not related to issues with protobuf. If I interrupt traced program execution early with a KeyboardInterrupt, the file opens without issues