The Cerebros package is an ultra-precise Neural Architecture Search (NAS) / AutoML that is intended to much more closely mimic biological neurons than conventional neural network architecture strategies.
Steps to reproduce the behavior: Intermittent with intended use.
Cerebros Enterprise Platform
If running on Kubeflow, which Jupyter notebook did you use as a template for your job? [N/A
Were you running this on public data that you can share? Can you provide an example of what the data looks like?
Ames Housing Data
Can you provide the code or combination of parameters that was involved (if not, its ok)? N/A
Can you copy and paste in any error or unexpected results?
2023-08-31 15:31:42.859762: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\nTraceback (most recent call last):\n File "[redacted].py", line [redacted], in [redacted]:
metric_prelim = cerebros_automl.run_random_search()
File "/opt/cerebros-core-algorithm-alpha/cerebros/simplecerebrosrandomsearch/simple_cerebros_random_search.py", line 524, in run_random_search
Process(target=self.run_moity_permutations(spec=spec, subtrial_number=subtrial_number, lock=lock)))
File "/opt/cerebros-core-algorithm-alpha/cerebros/simplecerebrosrandomsearch/simple_cerebros_random_search.py", line 478, in
run_moity_permutations
nnf.get_graph()
File "/opt/cerebros-core-algorithm-alpha/cerebros/neuralnetworkfuture/neural_network_future.py", line 431, in get_graph
net.save_graph(self.model_graph_file)
File "/usr/local/lib/python3.8/dist-packages/pyvis/network.py", line 435, in save_graph
self.write_html(name)
File "/usr/local/lib/python3.8/dist-packages/pyvis/network.py", line 533, in write_html
shutil.rmtree(f"{tempdir}/lib")\n File "/usr/lib/python3.8/shutil.py", line 722, in rmtree
onerror(os.rmdir, path, sys.exc_info())\n File "/usr/lib/python3.8/shutil.py", line 720, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: \'./lib\'
Expected behavior
It is unknown why this raised this exception selectively in this case. Additionally, the directory is usually not empty, and it does not throw this error.
Screenshots
N/A
Execution environment:
Linux: Modified tensorflow/tensorflow:2.12.0 container running on K8s.
Desktop (please complete the following information): N/A
Add any other context about the problem here.
The network graph visualization was a great tool for R and D purposes. It may be best to make this functionality optional in the Cerebros NAS core API and create a separate implementation that runs in a separate environment from the training task, or make it a separate module altogether.
This may need amended error handling in any case.
Suggested Labels (If you don't know, that's ok): kind/bug
Kind of issue: bug
Steps to reproduce the behavior: Intermittent with intended use.
Cerebros Enterprise Platform
If running on Kubeflow, which Jupyter notebook did you use as a template for your job? [N/A
Were you running this on public data that you can share? Can you provide an example of what the data looks like? Ames Housing Data
Can you provide the code or combination of parameters that was involved (if not, its ok)? N/A
Can you copy and paste in any error or unexpected results?
Expected behavior
It is unknown why this raised this exception selectively in this case. Additionally, the directory is usually not empty, and it does not throw this error.
Screenshots
N/A
Execution environment:
Linux: Modified tensorflow/tensorflow:2.12.0 container running on K8s.
Desktop (please complete the following information): N/A
Add any other context about the problem here.
Suggested Labels (If you don't know, that's ok): kind/bug