OSError: [Errno 9] Bad file descriptor

[caufieldjh]

When running the below config, the error OSError: [Errno 9] Bad file descriptor is raised. See further below for stack trace.

name: "quick_neat"
description: "A Quick NEAT Run"
output_directory: quickstart_output

graph_data:
  graph:
    node_path: tests/resources/test_graphs/test_small_nodes.tsv
    edge_path: tests/resources/test_graphs/test_small_edges.tsv
    directed: False
    verbose: True
    nodes_column: 'id'
    node_list_node_types_column: 'category'
    default_node_type: 'biolink:NamedThing'
    sources_column: 'subject'
    destinations_column: 'object'
    default_edge_type: 'biolink:related_to'

embeddings:
  embedding_file_name: quickstart_embedding.csv
  embedding_history_file_name: quickstart_embedding_history.json
  node_embedding_params:
    node_embedding_method_name: CBOW # one of 'CBOW', 'GloVe', 'SkipGram', 'Siamese', 'TransE', 'SimplE', 'TransH', 'TransR'
    walk_length: 10 # typically 100 or so
    batch_size: 128 # typically 512? or more
    window_size: 4
    return_weight: 1.0  # 1/p
    explore_weight: 1.0  # 1/q
    iterations: 5 # typically 20

Trace:

Exception ignored in: <function Pool.__del__ at 0x7eff78d69ca0>
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 268, in __del__
    self._change_notifier.put(None)
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put
    self._writer.send_bytes(obj)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
OSError: [Errno 9] Bad file descriptor

The embeddings are still saved.

[caufieldjh]

This appears to be an issue with Tensorflow and MirroredStrategy: https://github.com/tensorflow/tensorflow/issues/50487

[caufieldjh]

Newer versions of ensmallen/embiggen solve this issue, or just set the use_mirrored_strategy param for embeddings or classifiers to False.