Keras VGG16 Model

[Karl-Keller]

I'm using ArcGIS Pro 2.3.3 and I would like to use a trained model based on VGG16. Of course, there is no VGG16 module in the Keras directory here -

...AppData\Local\Programs\ArcGIS\Pro\Resources\Raster\Functions\System\DeepLearning\Keras

Since all the Keras models are part of the Keras applications module, should that be installed in the environment? Should a copy of the vgg16.py file be added to the directory above?

Is there a general document that explains the ArcGIS approach to adding new modules and the trained models?

Thanks!

[hanhu7]

Hi @Karl-Keller the built-in python raster functions in the ArcGIS resource folder ...AppData\Local\Programs\ArcGIS\Pro\Resources\Raster\Functions\System\DeepLearning is synced with this repo whenever ArcGIS is released. The keras module in this repo demonstrates and gives a good example and start point on how to inference a keras model using this specific implementation of mask r-cnn https://github.com/matterport/Mask_RCNN The deep learning technology is nothing specific in terms of python so this workflow is built on top of raster's python raster function framework in ArcGIS. Note that the two deep learning tools, detect objects and classify pixels using deep learning, ask for an emd file (JSON format) to be the input. If the "InferenceFunction" property is missing in the emd file, by default those two GP tools use the python raster functions in the ArcGIS resource folder, ObjectDetector.py for detecting objects and ImageClassifer.py for classifying pixels. If you have a custom keras model, most probably you will need to write a custom python raster function to process your data and would point the "InferenceFunction" in the emd file to your python raster function module. But this repo's source code could be always a good reference when working on any custom python raster function. Regarding the documentation, the README of this repo has listed lots of resources and I think starting with an understanding of imagery's python raster function would be good.

[Karl-Keller]

I downloaded the pre-trained coco model from Matterport repo, version 2.0, mask_rcnn_coco.h5.

The .emd file points to:

ModelFile --> "...\mask_rcnn_coco.h5" ModelConfiguration --> "MaskRCNN" InferenceFunction --> "...\ObjectDetector.py" ModelType --> "ObjectDetection"

When this model definition is used in the Detect Objects Using Deep Learning tool, it throws an error looking for the Config file, Line 22, of the ArcGIS supplied MaskRCNN.py file found in the ...DeepLearning/Keras subdirectory.

Following lingtangraster here: https://github.com/Esri/raster-deep-learning/issues/5, I expanded the JSON definition for ModelConfiguration to include Config and Architecture pointers.

As suggested, I provided a pointer to coco.py using the Matterport directory .../samples/coco in the repo.

ModelConfiguration: -->

{ Name --> "MaskRCNN" Config --> "...\samples\coco\coco" Architecture --> "...\samples\coco\coco" }

Coco.py doesn't contain either the 'config' or the 'model' subclasses required that MaskRCNN then looks for.

In ...samples/coco/coco.py, the config class definition, line 71, is actually the subclass CocoConfig of the Config parent and consequently isn't recognized by line 32 of the MaskRCNN.py file. Roughly the same error appears looking for the model subclass - which is actually MaskRCNN as defined in the ../mrcnn/model.py file, line 1820.

So, from the files provided in the Matterport repo and the ArcGIS installation, it seems like this should be running but examining the files clearly reveal something missing, either in my understanding or the naming conventions.

Any further explanation you can provide is great, and thank you for the first response! I've used the mask_rccn_coco.h5 file already on another effort along with vgg16. In python, running trained models appears more straightforward than with the addition of the .emd file in ArcGIS.

[hanhu7]

Hi @Karl-Keller In this example python raster function implementation, the python raster function inference uses an instance of MaskRCNN (https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/model.py#L1820-L1826) and that instance has to be created in the python module specified by "Architecture" in your emd file and also has to be named 'model' "ModelConfiguration": { "Name":"MaskRCNN", "Architecture":".\mrcnn\spacenet", "Config":".\mrcnn\spacenet" },

The same applies to the config object.

if you go to spacenet.py (https://github.com/Esri/raster-deep-learning/blob/master/examples/keras/mask_rcnn/mrcnn/spacenet.py) you can see there are the two python instances

config = InferenceConfig() model = modellib.MaskRCNN(mode='inference', config=config, model_dir='./')

The coco.py has the definition of CocoConfig but doesn't have an instance created and named 'config'. It does not have a MaskRCNN instance and named 'model' either.

Here is the code where the python raster function loads the model object created in the 'Architecture' python module. https://github.com/Esri/raster-deep-learning/blob/master/python_raster_functions/Keras/MaskRCNN.py#L61-L71

And here is the code where the model object is used for inference in the python raster function. https://github.com/Esri/raster-deep-learning/blob/master/python_raster_functions/Keras/MaskRCNN.py#L111-L117

A .h5 only has a model's weights in this case. To load the weights correctly, a python model object has to be created and imported from outside with some tolerance of changes in the model and config. Hope this explanation helps.

[Karl-Keller]

Thanks! Lots of information to dig through - hope I can get it going.

On Jun 27, 2019, at 5:45 PM, Han Hu notifications@github.com<mailto:notifications@github.com> wrote:

Hi @Karl-Kellerhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FKarl-Keller&data=02%7C01%7C%7C5c59a468fc9840cac64608d6fb48c4a2%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636972687283760068&sdata=creRJKMR7MUP6xpvoxuC8ZrI%2BEsOf%2FPZDV6nHOYCN%2BI%3D&reserved=0 In this example python raster function implementation, the python raster function inference uses an instance of MaskRCNN (https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/model.py#L1820-L1826https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmatterport%2FMask_RCNN%2Fblob%2Fmaster%2Fmrcnn%2Fmodel.py%23L1820-L1826&data=02%7C01%7C%7C5c59a468fc9840cac64608d6fb48c4a2%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636972687283770079&sdata=IeRZ7jdHYB1I96WjRFbtJFdl3dQOhyOwVD6fdeRyfTA%3D&reserved=0) and that instance has to be created in the python module specified by "Architecture" in your emd file and also has to be named 'model' "ModelConfiguration": { "Name":"MaskRCNN", "Architecture":".\mrcnn\spacenet", "Config":".\mrcnn\spacenet" },

The same applies to the config object.

if you go to spacenet.py (https://github.com/Esri/raster-deep-learning/blob/master/examples/keras/mask_rcnn/mrcnn/spacenet.pyhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEsri%2Fraster-deep-learning%2Fblob%2Fmaster%2Fexamples%2Fkeras%2Fmask_rcnn%2Fmrcnn%2Fspacenet.py&data=02%7C01%7C%7C5c59a468fc9840cac64608d6fb48c4a2%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636972687283780090&sdata=fLWsvtSZkYeu6vLYPEiAiWu0VTol5nN59kwZovIta2k%3D&reserved=0) you can see there are the two python instances

config = InferenceConfig() model = modellib.MaskRCNN(mode='inference', config=config, model_dir='./')

The coco.py has the definition of CocoConfig but doesn't have an instance created and named 'config'. It does not have a MaskRCNN instance and named 'model' either.

Here is the code where the python raster function loads the model object created in the 'Architecture' python module. https://github.com/Esri/raster-deep-learning/blob/master/python_raster_functions/Keras/MaskRCNN.py#L61-L71https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEsri%2Fraster-deep-learning%2Fblob%2Fmaster%2Fpython_raster_functions%2FKeras%2FMaskRCNN.py%23L61-L71&data=02%7C01%7C%7C5c59a468fc9840cac64608d6fb48c4a2%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636972687283780090&sdata=4pj1vomsL2WHqXHbQB2pXYQR7iMRxYVvQs%2Bk0FJOtLo%3D&reserved=0

And here is the code where the model object is used for inference in the python raster function. https://github.com/Esri/raster-deep-learning/blob/master/python_raster_functions/Keras/MaskRCNN.py#L111-L117https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEsri%2Fraster-deep-learning%2Fblob%2Fmaster%2Fpython_raster_functions%2FKeras%2FMaskRCNN.py%23L111-L117&data=02%7C01%7C%7C5c59a468fc9840cac64608d6fb48c4a2%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636972687283790102&sdata=yozivkzkhHYH%2F2rAZirAnXjkERMf8HbvSZWJNT0pdQQ%3D&reserved=0

A .h5 only has a model's weights in this case. To load the weights correctly, a python model object has to be created and imported from outside with some tolerance of changes in the model and config. Hope this explanation helps.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEsri%2Fraster-deep-learning%2Fissues%2F18%3Femail_source%3Dnotifications%26email_token%3DAHWQG566TUG26OLKT23WZC3P4UYHLA5CNFSM4H27TRA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYYO5IQ%23issuecomment-506523298&data=02%7C01%7C%7C5c59a468fc9840cac64608d6fb48c4a2%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636972687283800107&sdata=zaD7WpsVstuWXHDAXKx7bMdJhpz4oysABgj0d%2BJs4ds%3D&reserved=0, or mute the threadhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHWQG52IXTPGUP7HRQAG3L3P4UYHLANCNFSM4H27TRAQ&data=02%7C01%7C%7C5c59a468fc9840cac64608d6fb48c4a2%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636972687283810118&sdata=vR5QkOxxxAmjf%2F71FK0T8rUmNsJA31%2FIApJ9ptLzCf4%3D&reserved=0.

[Karl-Keller]

Thanks for the help - it got me past the config/model error.

Although, coco.py does conditionally define subclass InferenceConfig (line 443) and then instantiate InferenceConfig (line 449) and MaskRCNN (lines 454/457 switched via 'training' or 'inference').

Regardless, I instantiated Config and MaskRCNN as config and model as you indicated using my own net.py file in 'inference' mode.

This dropped me into another error (valerror, shape mismatch) that I'll document in a new thread.