search

conservationtechlab / animl-r

Animl comprises a variety of machine learning tools for analyzing ecological data. The package includes a set of functions to classify subjects within camera trap field data and can handle both images and videos.
MIT License
19 stars 5 forks source link

image classifcation function throws error with corrupt images files #3

Open matobler opened 2 years ago

matobler commented 2 years ago

When an image file is corrupted (e.g. was not terminated) we get the following error message from the classifySpecies function:

`2022-03-24 14:24:32.329992: W tensorflow/core/framework/op_kernel.cc:1680] Unknown: OSError: image file is truncated (35 bytes not processed) Traceback (most recent call last):

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in call ret = func(*args)

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 645, in wrapper return func(*args, **kwargs)

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 892, in generator_py_func values = next(generator_state.get_iterator(iterator_id))

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\keras\engine\data_adapter.py", line 822, in wrapped_generator for data in generator_fn():

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\keras\engine\data_adapter.py", line 948, in generator_fn yield x[i]

File "", line 109, in getitem

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\PIL\Image.py", line 1146, in crop self.load()

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\PIL\ImageFile.py", line 248, in load "image file is truncated "

OSError: image file is truncated (35 bytes not processed)

Error in py_call_impl(callable, dots$args, dots$keywords) : UnknownError: 2 root error(s) found. (0) Unknown: OSError: image file is truncated (35 bytes not processed) Traceback (most recent call last):

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in call ret = func(*args)

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 645, in wrapper return func(*args, **kwargs)

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 892, in generator_py_func values = next(generator_state.get_iterator(iterator_id))

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\keras\engine\data_adapter.py", line 822, in wrapped_generator for data in generator_fn():

File "C:\Users\mtobler\ANACON~1\envs\TE1D75~1\lib\site-packages\keras\engine\data_adapter.py", line 948, in generator_fn yield x[i]

File "", line 109, In addition: Warning message: In predict_generator(., generator = cropGenerator, steps = predict_steps, : `

Need to add error handling for this.

tkswanson commented 2 years ago

Ive tried, its really hard to do with the image generator. I can't seem to find the right place to catch the error. I ended up just allowing truncated images in the current version...

On Thu, Mar 24, 2022, 15:38 Mathias Tobler @.***> wrote:

When an image file is corrupted (e.g. was not terminated) we get the following error message from the classifySpecies function:

`2022-03-24 14:24:32.329992: W tensorflow/core/framework/op_kernel.cc:1680] Unknown: OSError: image file is truncated (35 bytes not processed) Traceback (most recent call last):

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in call ret = func(*args)

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 645, in wrapper return func(*args, **kwargs)

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 892, in generator_py_func values = next(generator_state.get_iterator(iterator_id))

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 822, in wrapped_generator for data in generator_fn():

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 948, in generator_fn yield x[i]

File "", line 109, in getitem

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\PIL\Image.py", line 1146, in crop self.load()

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\PIL\ImageFile.py", line 248, in load "image file is truncated "

OSError: image file is truncated (35 bytes not processed)

Error in py_call_impl(callable, dots$args, dots$keywords) : UnknownError: 2 root error(s) found. (0) Unknown: OSError: image file is truncated (35 bytes not processed) Traceback (most recent call last):

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in call ret = func(*args)

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 645, in wrapper return func(*args, **kwargs)

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 892, in generator_py_func values = next(generator_state.get_iterator(iterator_id))

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 822, in wrapped_generator for data in generator_fn():

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 948, in generator_fn yield x[i]

File "", line 109, In addition: Warning message: In predict_generator(., generator = cropGenerator, steps = predict_steps, : `

Need to add error handling for this.

— Reply to this email directly, view it on GitHub https://github.com/icr-ctl/animl/issues/3, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBLJ4ST5JDML6R3HL4HXP3VBTVH5ANCNFSM5RSP7LZQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

matobler commented 2 years ago

This particular issue was likely solved with the new tf.data generator, but there might still be some cases that could lead to errors. Currently I check for zero-length files and the decoder function can handle truncated files. But there might be other possible corruptions.