Closed MartinKjunior closed 6 months ago
I have updated all the libraries and replaced the np.int(), but keep getting the error message "Fatal Python error: Illegal Instruction". Would you know what the problem is?
I have updated all the libraries and replaced the np.int(), but keep getting the error message "Fatal Python error: Illegal Instruction". Would you know what the problem is?
I have not encountered this issue. Could you provide full traceback as well as the individual lines you've changed from the original?
I only changed np.int() on line 227,263 and 276. This is the full error:
I only changed np.int() on line 227,263 and 276. This is the full error:
Hm... I'm not sure what the problem might be. Are you able to run other tensorflow-based scripts? Check that the way you've set up your environment is okay. I think I'm also running python 3.11 and I've run the script on both Windows and Linux using the tensorflow-cpu version.
The code does not work when I run it on Mac, but Windows works! I however now I get the Error: ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (4,2) and requested shape (3,2)
The code does not work when I run it on Mac, but Windows works! I however now I get the Error: ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (4,2) and requested shape (3,2)
Ah so it was a mac issue then. Maybe something to do with their new chips? Don't know.
Are you using a 3D image? Also please post full traceback whenever you share an error, thanks.
Yes I think the new chip is not compatible with Tensorflow. I am using a 2D stack of 181 images. The full error is:
runfile('C:/Users/scanimage/Desktop/3p V/Protocoles/DeepVess/DeepVess.py', wdir='C:/Users/scanimage/Desktop/3p V/Protocoles/DeepVess') WARNING:tensorflow:From C:\ProgramData\Anaconda3\Lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.
WARNING:tensorflow:From c:\users\scanimage\desktop\3p v\protocoles\deepvess\deepvess.py:63: The name tf.disable_eager_execution is deprecated. Please use tf.compat.v1.disable_eager_execution instead.
Traceback (most recent call last):
File C:\ProgramData\Anaconda3\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec exec(code, globals, locals)
File c:\users\scanimage\desktop\3p v\protocoles\deepvess\deepvess.py:87 im = np.pad(im, padSize, 'symmetric')
File <__array_function__ internals>:200 in pad
File C:\ProgramData\Anaconda3\Lib\site-packages\numpy\lib\arraypad.py:744 in pad pad_width = _as_pairs(pad_width, array.ndim, as_index=True)
File C:\ProgramData\Anaconda3\Lib\site-packages\numpy\lib\arraypad.py:518 in _as_pairs return np.broadcast_to(x, (ndim, 2)).tolist()
File <__array_function__ internals>:200 in broadcast_to
File C:\ProgramData\Anaconda3\Lib\site-packages\numpy\lib\stride_tricks.py:413 in broadcast_to return _broadcast_to(array, shape, subok=subok, readonly=True)
File C:\ProgramData\Anaconda3\Lib\site-packages\numpy\lib\stride_tricks.py:349 in _broadcast_to it = np.nditer(
ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (4,2) and requested shape (3,2)
Yes I think the new chip is not compatible with Tensorflow. I am using a 2D stack of 181 images. The full error is:
runfile('C:/Users/scanimage/Desktop/3p V/Protocoles/DeepVess/DeepVess.py', wdir='C:/Users/scanimage/Desktop/3p V/Protocoles/DeepVess') WARNING:tensorflow:From C:\ProgramData\Anaconda3\Lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.
WARNING:tensorflow:From c:\users\scanimage\desktop\3p v\protocoles\deepvess\deepvess.py:63: The name tf.disable_eager_execution is deprecated. Please use tf.compat.v1.disable_eager_execution instead.
Traceback (most recent call last):
File C:\ProgramData\Anaconda3\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec exec(code, globals, locals)
File c:\users\scanimage\desktop\3p v\protocoles\deepvess\deepvess.py:87 im = np.pad(im, padSize, 'symmetric')
File <array_function internals>:200 in pad
File C:\ProgramData\Anaconda3\Lib\site-packages\numpy\lib\arraypad.py:744 in pad pad_width = _as_pairs(pad_width, array.ndim, as_index=True)
File C:\ProgramData\Anaconda3\Lib\site-packages\numpy\lib\arraypad.py:518 in _as_pairs return np.broadcast_to(x, (ndim, 2)).tolist()
File <array_function internals>:200 in broadcast_to
File C:\ProgramData\Anaconda3\Lib\site-packages\numpy\lib\stride_tricks.py:413 in broadcast_to return _broadcast_to(array, shape, subok=subok, readonly=True)
File C:\ProgramData\Anaconda3\Lib\site-packages\numpy\lib\stride_tricks.py:349 in _broadcast_to it = np.nditer(
ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (4,2) and requested shape (3,2)
First thing I wanted to mention was that make sure your images are similar, especially when it comes to the background signal. I used this script to segment a time series of 3D images of blood vessels after injecting tracers that leaked a bit by 1. finding the MIP to get a 3D stack and 2. individually segmenting the 3D images and I think it does a better job on individual images when the background signal changes a lot.
When it comes to the error, check what the shape of im = np.array(f.get('/im'))
is after your image gets loaded (line 72, throw in a print(f"{im.shape=}")
right after that line) and check that it's definitely the correct shape. I have always used a single 3D image which I saved as TIFF before running it through the preprocessing steps in matlab (I assume you used the matlab scripts to preprocess) and then fed that into the python script.
I found the problem, the pre-process code in Matlab was only reading the first image of the 3D stack making it a 2D image. That works now, but the checkpoint saving does not work. Do you know what the problem is? This is the error: runfile('C:/Users/scanimage/Desktop/3p V/Protocoles/DeepVess/DeepVess.py', wdir='C:/Users/scanimage/Desktop/3p V/Protocoles/DeepVess') im.shape=(88, 512, 512) Data loaded. Traceback (most recent call last):
File C:\ProgramData\Anaconda3\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec exec(code, globals, locals)
File c:\users\scanimage\desktop\3p v\protocoles\deepvess\deepvess.py:254 saver.restore(sess, os.path.join('private', 'model-epoch29999.ckpt'))
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\training\saver.py:1410 in restore raise ValueError("The passed save_path is not a valid checkpoint: " +
ValueError: The passed save_path is not a valid checkpoint: private\model-epoch29999.ckpt
I found the problem, the pre-process code in Matlab was only reading the first image of the 3D stack making it a 2D image. That works now, but the checkpoint saving does not work. Do you know what the problem is? This is the error: runfile('C:/Users/scanimage/Desktop/3p V/Protocoles/DeepVess/DeepVess.py', wdir='C:/Users/scanimage/Desktop/3p V/Protocoles/DeepVess') im.shape=(88, 512, 512) Data loaded. Traceback (most recent call last):
File C:\ProgramData\Anaconda3\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec exec(code, globals, locals)
File c:\users\scanimage\desktop\3p v\protocoles\deepvess\deepvess.py:254 saver.restore(sess, os.path.join('private', 'model-epoch29999.ckpt'))
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\training\saver.py:1410 in restore raise ValueError("The passed save_path is not a valid checkpoint: " +
ValueError: The passed save_path is not a valid checkpoint: private\model-epoch29999.ckpt
In my case that had to do with the cwd of the script that was running. I fixed it by adding one more folder to saver.restore(sess, os.path.join('DeepVess','private', 'model-epoch29999.ckpt'))
. Essentially you want to make sure that the second argument in saver.restore()
is a relative path pointing to the saved model (I haven't tried if using absolute path works as well).
That worked thanks! Now its blocking saying it blocks at Key Variable_100, which I don't understand: runfile('C:/Users/scanimage/Desktop/3p V/Protocoles/DeepVess/DeepVess.py', wdir='C:/Users/scanimage/Desktop/3p V/Protocoles/DeepVess')
C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py:1793: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call InteractiveSession.close()
to release resources held by the other session(s).
warnings.warn('An interactive session is already active. This can '
im.shape=(88, 512, 512)
Data loaded.
INFO:tensorflow:Restoring parameters from private\model-epoch29999.ckpt
Traceback (most recent call last):
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py:1402 in _do_call return fn(*args)
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py:1385 in _run_fn return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py:1478 in _call_tf_sessionrun return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
NotFoundError: Key Variable_100 not found in checkpoint [[{{node save_8/RestoreV2}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\training\saver.py:1418 in restore sess.run(self.saver_def.restore_op_name,
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py:972 in run result = self._run(None, fetches, feed_dict, options_ptr,
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py:1215 in _run results = self._do_run(handle, final_targets, final_fetches,
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py:1395 in _do_run return self._do_call(_run_fn, feeds, fetches, targets, options,
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py:1421 in _do_call raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
NotFoundError: Graph execution error:
Detected at node 'save_8/RestoreV2' defined at (most recent call last):
File "
Original stack trace for 'save_8/RestoreV2':
File "
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py:66 in get_tensor return CheckpointReader.CheckpointReader_GetTensor(
RuntimeError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\training\saver.py:1429 in restore names_to_keys = object_graph_key_mapping(save_path)
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\training\saver.py:1751 in object_graph_key_mapping object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY)
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py:71 in get_tensor error_translator(e)
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py:31 in error_translator raise errors_impl.NotFoundError(None, None, error_message)
NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File C:\ProgramData\Anaconda3\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec exec(code, globals, locals)
File c:\users\scanimage\desktop\3p v\protocoles\deepvess\deepvess.py:254 saver.restore(sess, os.path.join('private', 'model-epoch29999.ckpt'))
File C:\ProgramData\Anaconda3\Lib\site-packages\tensorflow\python\training\saver.py:1434 in restore raise _wrap_restore_error_with_msg(
NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Graph execution error:
Detected at node 'save_8/RestoreV2' defined at (most recent call last):
File "
Original stack trace for 'save_8/RestoreV2':
File "
2024-03-21 12:02:43.644493: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at save_restore_v2_ops.cc:233 : NOT_FOUND: Key Variable_100 not found in checkpoint
@elinestas I'm sorry I have no idea how to solve this and I have not met this error before. Quick google search says the issue is with the model having something that is absent in the saved model. I have not changed the model in any way and did not attempt to retrain it for my dataset.
@MartinKjunior, thanks for submitting this issue, I hope the patch fixed the issue.
@elinestas, feel free to send me an email, and I would be more than happy to get DeepVess up and running ASAP.
The file DeepVess.py uses the deprecated
np.int()
, but replacing the two instances withint()
allows the script to run.