tensorflow version - NEON.h5 (file signature not found)

[aloui-mathias]

Hi, Since the Pytorch-Release, I'am having issue with the tesorflow version. I tried using my script to make a prediction but when loading the model I had an error. I tried to reload the model by changing the name in current_release.csv to force a new download but still the same error. Thankfully I have an old backup virtual environment. I replace the NEON.h5 file with the one from this old env and it worked.

Here is an example of my error :

In [1]: from deepforest import deepforest Using TensorFlow backend.

In [2]: model = deepforest.deepforest() Reading config file: C:\Users\MA\anaconda3\envs\DeepForestTourne\lib\site-packages\deepforest\data\deepforest_config.yml A blank deepforest object created. To perform prediction, either train or load an existing model.

In [3]: model.use_release() Model from DeepForest release https://github.com/weecology/DeepForest/releases/tag/1.0.0 was already downloaded. Loading model from file. Loading pre-built model: https://github.com/weecology/DeepForest/releases/tag/1.0.0 C:\Users\MA\anaconda3\envs\DeepForestTourne\lib\site-packages\deepforest\data/NEON.h5

OSError Traceback (most recent call last)

in ----> 1 model.use_release() ~\anaconda3\envs\DeepForestTourne\lib\site-packages\deepforest\deepforest.py in use_release(self, gpus) 169 # Suppress compilte warning, not relevant here 170 warnings.filterwarnings("ignore", category=UserWarning) --> 171 self.model = utilities.read_model(self.weights, self.config) 172 173 # Convert model ~\anaconda3\envs\DeepForestTourne\lib\site-packages\deepforest\utilities.py in read_model(model_path, config) 48 # warnings.simplefilter('ignore', UserWarning) 49 print(model_path) ---> 50 model = models.load_model(model_path, backbone_name='resnet50') 51 52 return model ~\anaconda3\envs\DeepForestTourne\lib\site-packages\deepforest\keras_retinanet\models\__init__.py in load_model(filepath, backbone_name) 81 """ 82 import keras.models ---> 83 return keras.models.load_model(filepath, custom_objects=backbone(backbone_name).custom_objects) 84 85 ~\anaconda3\envs\DeepForestTourne\lib\site-packages\keras\engine\saving.py in load_wrapper(*args, **kwargs) 490 os.remove(tmp_filepath) 491 return res --> 492 return load_function(*args, **kwargs) 493 494 return load_wrapper ~\anaconda3\envs\DeepForestTourne\lib\site-packages\keras\engine\saving.py in load_model(filepath, custom_objects, compile) 581 582 if H5Dict.is_supported_type(filepath): --> 583 with H5Dict(filepath, mode='r') as h5dict: 584 model = _deserialize_model(h5dict, custom_objects, compile) 585 elif hasattr(filepath, 'write') and callable(filepath.write): ~\anaconda3\envs\DeepForestTourne\lib\site-packages\keras\utils\io_utils.py in __init__(self, path, mode) 189 self._is_file = False 190 elif isinstance(path, six.string_types) or _is_path_instance(path): --> 191 self.data = h5py.File(path, mode=mode) 192 self._is_file = True 193 elif isinstance(path, dict): ~\anaconda3\envs\DeepForestTourne\lib\site-packages\h5py\_hl\files.py in __init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, **kwds) 406 fid = make_fid(name, mode, userblock_size, 407 fapl, fcpl=make_fcpl(track_order=track_order), --> 408 swmr=swmr) 409 410 if isinstance(libver, tuple): ~\anaconda3\envs\DeepForestTourne\lib\site-packages\h5py\_hl\files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr) 171 if swmr and swmr_support: 172 flags |= h5f.ACC_SWMR_READ --> 173 fid = h5f.open(name, flags, fapl=fapl) 174 elif mode == 'r+': 175 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl) h5py\_objects.pyx in h5py._objects.with_phil.wrapper() h5py\_objects.pyx in h5py._objects.with_phil.wrapper() h5py\h5f.pyx in h5py.h5f.open() OSError: Unable to open file (file signature not found)

[bw4sz]

Thanks. I am making a document today about how best to update. I think the cleanest thing is to manually load the desired tensorflow release model.

Download here and then load weights

reloaded = deepforest.deepforest(weights="example_save_weights.h5")

https://github.com/weecology/DeepForest/releases/tag/v0.3.0

I just had another user report that the performance was different than expected between the use_release() and the download model. This is extremely surprising since they are the same model. #195

Please let me know if you experience any change in performance. Close when ready.

[aloui-mathias]

Thanks for the quick response.

Yes I also have a problem of performance with the pytorch version. I'm using DeepForest in urban area (and I'm going to train it on new urban data to have a better model) and the tensorflow version is pretty good but the pytorch version is really not great and is very sensitive to changes in patch_size and overlap.

[bw4sz]

can you show some example images? The underlying data and algorithm is basically identical.

On Thu, Jun 10, 2021 at 4:23 AM Mathias Aloui @.***> wrote:

Thanks for the quick response.

Yes I also have a problem of performance with the pytorch version. I'm using DeepForest in urban area (and I'm going to train it on new urban data to have a better model) and the tensorflow version is pretty good but the pytorch version is really not great and is very sensitive to changes in patch_size and overlap.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/weecology/DeepForest/issues/198#issuecomment-858538447, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJHBLEFQ7BHJXBYVLP4K2LTSCOB7ANCNFSM46LV4CGQ .

-- Ben Weinstein, Ph.D. Postdoctoral Fellow University of Florida http://benweinstein.weebly.com/

[aloui-mathias]

Here is an example with the same image and the same parameters (patch_size = 400, patch_overlap = 0.15, iou_threshold=0.15) :

The first with tensorflow :

The second with pytorch :

Note : pytorch is way faster than tensorflow

[bw4sz]

can you pass me that raw image? This is surprising, there must be an additional parameter change somewhere, minimum score threshold between the models? We test against the benchmark here: https://github.com/weecology/NeonTreeEvaluation and the pytorch model is within 1% of the tensorflow of the model with no visible differences. The training data are the same between those two releases. I just retrained a model to add new annotations, including a small urban tile, I will upload. Would you be willing to share your annotations and i'll add them in for future users? I'm going to have a look at your image, but something doesn't quite make sense. I'm going to test it on predict_image first, to make sure it isn't some tiling method between the old and new versions. That seems more likely than the actual model weights.

On Thu, Jun 10, 2021 at 1:13 PM Mathias Aloui @.***> wrote:

Here is an example with the same image and the same parameters (patch_size = 400, patch_overlap = 0.15, iou_threshold=0.15) :

The first with tensorflow : [image: tensorflow] https://user-images.githubusercontent.com/43454650/121590723-f1e18480-ca38-11eb-9142-06317a06fb67.png

The second with pytorch : [image: pytorch] https://user-images.githubusercontent.com/43454650/121590722-f148ee00-ca38-11eb-8a7e-3d8d79d37aed.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/weecology/DeepForest/issues/198#issuecomment-859012783, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJHBLBTPT35MQVHV7CAGDTTSEMHTANCNFSM46LV4CGQ .

-- Ben Weinstein, Ph.D. Postdoctoral Fellow University of Florida http://benweinstein.weebly.com/

[aloui-mathias]

Here is the image I used :

I didn't check if other parameters were different (I assumed that all default parameters were the same).

The annotations I have are for the challenge Detect Trees for the Hack4Nature (https://hack4nature.slite.com/p/note/Fs34nEyzDG61edEM5oHnUF/Challenge-4-Detect-Tree). We have a wiki Slite with everything we use and soon we will put a link to a web page for online annotations and all the data we use. I can keep you posted on our advancement ;-)

[bw4sz]

Great, thanks, keep my posted. I'm downloading that image and playing with it. I just trained a new model. I really can't imagine the tensorflow and pytorch versions should be meaningfully different. It must be in preprocessing. I'll update.

On Fri, Jun 11, 2021 at 12:04 AM Mathias Aloui @.***> wrote:

Here is the image I used :

[image: image] https://user-images.githubusercontent.com/43454650/121643179-d191e400-ca91-11eb-88d0-011a9b8b3e52.png

I didn't check if other parameters were different (I assumed that all default parameters were the same).

The annotations I have are for the challenge Detect Trees for the Hack4Nature (https://www.hackfornature.com/blog/challenge-4-detect-tree http://url). We have a wiki Slite with everything we use and soon we will put a link to a web page for online annotations and all the data we use. I can keep you posted on our advancement ;-)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/weecology/DeepForest/issues/198#issuecomment-859333834, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJHBLABPEGD4D3N6TSAQEDTSGYRHANCNFSM46LV4CGQ .

-- Ben Weinstein, Ph.D. Postdoctoral Fellow University of Florida http://benweinstein.weebly.com/

[bw4sz]

Just posting a few sample images as a try to reproduce

from deepforest import main
from matplotlib import pyplot as plt
m=main.deepforest()
m.use_release()
a = m.predict_image(path = "/Users/benweinstein/Downloads/test_image.png",return_plot=True)
plt.imshow(a[:,:,::-1])

[aloui-mathias]

Just a little follow-up on the performance issue :

I saw your pretty good prediction with predict_image and I wondered what about the evaluation methode. I tested the evaluation methode on the prediction from predict_tile and predict_image. For predict_image everything is perfect, everything at 100%. But for predict_tile it's a mess. class precision and recall are 100% but box recall and precision are not. I think like you said there is a problem with the preprocessing, with the tiling process.

A comment on how evaluation works :

Right now the evaluation method main.deepforest.evaluate works like predict_image to make the prediction. Which explains why everything works on predict_image predictions and not on predict_tile predictions. If the preprocessing is fixed, we will still need an evaluation method that works like predict_tile to make prediction otherwise for big image the evaluation will not be accurate.

My own prediction :

I used the exact same code you used and I have a slightly different prediction, strange ?

My tests :

Here is the notebook I used for my evaluation tests

pytorch.zip

Edit :

I found out that changing the patch_size change drastically the performance for pytorch predict_tile

[bw4sz]

Changing patch_size will definitely have performance for predict tile, that's a feature not a bug. For every input resolution, there will be an optimal patch size. There is no way to anticipate for all users. The question is why did earlier versions of deepforest predict_tile work better for you? I will check the default patch size and try to see what has changed.

I cannot fathom why using the exact same code would produce different results. That seems like an important thing to investigate. Thanks for following up.

[bw4sz]

I'm still trying to wrap my head around this. I think you are not using the code quite as intended. Let's figure out how to make the docs better. The evaluate method is meant for when you have a .csv of annotations for each image. You are 100% right there is not an evaluate method directly for a tile. You would need to cut the tile into pieces using preprocess.split_raster, save the output and then evaluate on this. I can make an example in the docs. Maybe make a quick video in evaluate.

[bw4sz]

Evaluating tiles

The evaluation method uses deepforest.predict_image for each of the paths supplied in the image_path column. This means that the entire image is passed for prediction. This will not work for large orthomosaics. The deepforest.predict_tile method does a couple things under hood that need to be repeated for evaluation.

psuedo_code:

output_annotations = deepforest.preprocess.split_raster(
    path_to_raster = <path>,
    annotations_file = <original_annotation_path>,
    base_dir = <location to save crops>
    patch_size = <size of each crop>

)

output_annotations.to_csv("new_annotations.csv")

results = model.evaluate(
    csv_file="new_annotations.csv",
    root_dir=<base_dir from above>
)

[Mashood3624]

@bw4sz How to download weights for "Alive" and "Dead" labels ?

[Mashood3624]

model ****