search

aimhubio / aim

Aim 💫 — An easy-to-use & supercharged open-source experiment tracker.
https://aimstack.io
Apache License 2.0
5.16k stars 316 forks source link

Failed to execute 'json' on 'Response': body stream already read #3151

Closed jennifer12121 closed 4 months ago

jennifer12121 commented 4 months ago

🐛 Bug

Runs become hung in the in progress state and an error message flashes on the side "Failed to execute 'json' on 'Response': body stream already read". When this happens, the run is not able to be closed or moved to a closed state.

Additionally, from the terminal, when trying to run aim runs close hash an error message appears, so the runs are not able to be closed through the sdk or the cli.

`Traceback (most recent call last): File "aim/storage/treeutils.pyx", line 112, in aim.storage.treeutils.iter_fold_tree File "aim/storage/treeutils.pyx", line 201, in aim.storage.treeutils.DecodePathsVals.next File "aim/storage/treeutils.pyx", line 214, in aim.storage.treeutils.DecodePathsVals._next StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "aim/storage/containertreeview.py", line 71, in aim.storage.containertreeview.ContainerTreeView.collect File "aim/storage/treeutils.pyx", line 270, in aim.storage.treeutils.decode_tree File "aim/storage/treeutils.pyx", line 95, in aim.storage.treeutils.fold_tree File "aim/storage/treeutils.pyx", line 124, in iter_fold_tree KeyError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/aim", line 8, in sys.exit(cli_entry_point()) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 783, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, *kwargs) File "/usr/local/lib/python3.8/site-packages/aim/cli/runs/commands.py", line 159, in closeruns for in tqdm.tqdm( File "/usr/local/lib/python3.8/site-packages/tqdm/std.py", line 1181, in iter for obj in iterable: File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 868, in next raise value File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(args, kwds)) File "/usr/local/lib/python3.8/site-packages/aim/sdk/repo.py", line 976, in _close_run index_manager.index(run_hash) File "/usr/local/lib/python3.8/site-packages/aim/sdk/index_manager.py", line 175, in index if meta_run_tree['end_time'] is None: File "aim/storage/treeview.py", line 51, in aim.storage.treeview.TreeView.getitem File "aim/storage/containertreeview.py", line 73, in aim.storage.containertreeview.ContainerTreeView.collect KeyError: "No key ('end_time',) is present."`

To reproduce

Attempt to track multiple runs to Aim. They will not close out and will hang in progress.

Expected behavior

It is expected to close the runs either upon failure or upon completion.

Environment

jennifer12121 commented 4 months ago

Upgrading to 3.19.3 resolved this.