Error during training : AttributeError: 'Tee' object has no attribute 'close'

[Addalin]

To reproduce - run main_lightining.py

(pid=28740) Error in atexit._run_exitfuncs:
(pid=28740) Traceback (most recent call last):
(pid=28740)   File "C:\Users\addalin\.conda\envs\lidar\lib\logging\__init__.py", line 2123, in shutdown
2021-05-08 05:14:36,713 INFO tune.py:549 -- Total run time: 122255.10 seconds (122254.85 seconds for the tuning loop).
(pid=28740)     h.close()
(pid=28740)   File "C:\Users\addalin\.conda\envs\lidar\lib\site-packages\absl\logging\__init__.py", line 945, in close
(pid=28740)     self.stream.close()
(pid=28740) AttributeError: 'Tee' object has no attribute 'close'

[Addalin]

(pid=23996) Traceback (most recent call last):
(pid=23996)   File "C:\Users\addalin\.conda\envs\lidar\lib\logging\__init__.py", line 2123, in shutdown
(pid=23996)     h.close()
(pid=23996)   File "C:\Users\addalin\.conda\envs\lidar\lib\site-packages\absl\logging\__init__.py", line 945, in close
(pid=23996)     self.stream.close()
(pid=23996) AttributeError: 'Tee' object has no attribute 'close'
Traceback (most recent call last):
  File "C:/Users/addalin/Dropbox/Lidar/lidar_learning/learning_lidar/learning_phase/main_lightning.py", line 81, in <module>
    analysis = tune.run(
  File "C:\Users\addalin\.conda\envs\lidar\lib\site-packages\ray\tune\tune.py", line 543, in run
    raise TuneError("Trials did not complete", incomplete_trials)
ray.tune.error.TuneError: ('Trials did not complete', [main_23b39_00000, main_23b39_00007])

Process finished with exit code 1```

@scaperex  this issue keeps repeating. 
Though it can be fixed after with the 'RESUME_EXP' option,  it seems that we are missing out on something in here. And it's annoying:-\

Can you please take a look at this (next week)?

[OmerShubi]

This seems to be related to ray or ray+lighting and to the way they perform the logging. (I removed all instances of our logging and still reproduced the issue) It could be useful to update ray, and see if this issue still occurs. If it still occurs let me know and I'll open an issue on Ray's repo.

[Addalin]

After updating to ray 1.5.2, issue does not appear. Replaced with: See ray issue