Training step error

[zehrea]

Hello,

I am getting error below during training step. What could be the issue?

Many thanks, Elena

cryoCARE_train.py --conf train_config.json /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) Using TensorFlow backend. /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/csbdeep/models/base_model.py:148: UserWarning: output path for model already exists, files may be overwritten: /gpfs/gsfs11/users/zehrea/data/cryo_data/TOMOGRAPHY/Microtubules/cryoCare/model_name warnings.warn('output path for model already exists, files may be overwritten: %s' % str(self.logdir.resolve())) WARNING:tensorflow:From /usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, there are two options available in V2.

• tf.py_function takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means tf.py_functions can use accelerators such as GPUs as well as being differentiable using a gradient tape.
• tf.numpy_function maintains the semantics of the deprecated tf.py_func (it is not differentiable, and manipulates numpy arrays). It drops the stateful argument making all functions stateful.

Traceback (most recent call last): File "/usr/local/apps/cryocare_pip/conda/envs/cryocare/bin/cryoCARE_train.py", line 43, in main() File "/usr/local/apps/cryocare_pip/conda/envs/cryocare/bin/cryoCARE_train.py", line 36, in main history = model.train(dm.get_train_dataset(), dm.get_val_dataset()) File "/usr/local/apps/cryocare_pip/conda/envs/cryocare/lib/python3.7/site-packages/cryocare/internals/CryoCARE.py", line 33, in train axes = axes_check_and_normalize('S' + self.config.axes, len(train_dataset.element_spec[0].shape) + 1) AttributeError: 'DatasetV1Adapter' object has no attribute 'element_spec'

[zehrea]

This is the header of my file:

RO image file on unit 1 : odd/odd_rec.mrc Size= 2234376 K

Number of columns, rows, sections ..... 2860 4000 200 Map mode .............................. 0 (bytes - signed in file)
Start cols, rows, sects, grid x,y,z ... 0 0 0 2860 4000 200 Pixel spacing (Angstroms).............. 5.040 5.040 5.040
Cell angles ........................... 90.000 90.000 90.000 Fast, medium, slow axes ............... X Y Z Origin on x,y,z ..(inverted_infile).. -2873. 504.0 2873.
Minimum density ....................... 0.0000 ( -128.00 in file) Maximum density ....................... 255.00 ( 127.00 in file) Mean density .......................... 162.41 ( 34.415 in file) tilt angles (original,current) ........ 0.0 0.0 0.0 90.0 0.0 0.0 Space group,# extra bytes,idtype,lens . 1 0 0 0

[thorstenwagner]

The last error message tells me that this is the same issue as issue duplicate issue https://github.com/juglab/cryoCARE_pip/issues/13 . Can you try the solutions suggested there ? I will close this for now.

[zehrea]

Hello, thanks for your response.

Is there a comprehensive list of dependencies for cryoCARE that includes tensorflow 2.3.0? The problem is that very little is defined, and what is defined is somewhat old. The current version of tensorflow is 2.9.

Elena