search

mintproject / mic

Model Insertion Checker
https://mic-cli.readthedocs.io/en/latest/
5 stars 3 forks source link

Issue with cwltool jupyter notebook #344

Closed iperezx closed 3 years ago

iperezx commented 3 years ago

Describe the bug Can't managed to get this part in the user guide with my own repo: https://mic-cli.readthedocs.io/en/latest/notebooks/run_component/#run-and-test-your-component I have follow the previous instructions succesfully. I am able to run the example notebook just fine.

I am confused of why it is complaining about one of my custom modules that I am importing since in the binder environment it doesn't complain about any missing modules. I am assuming that if it finds the module then it will not complain about the output file (the other error).

My binder instance for reference: https://mybinder.org/v2/gh/iperezx/edge-plugin-smokedetect/3fcd9a3bc119c818af99fb184ec21a83bc2318a3?filepath=main.ipynb

Any clues?

Debugging:

I tried to add more flags to better debug the issue with cwltool: cwltool --leave-tmpdir --leave-outputs --leave-container --debug main.cwl modelValues.yml

I ran the image that was generated by repo2docker: r2d-2fvar-2ffolders-2fj6-2ff847737s17s4hwjc1g8n46600000gn-2ft-2frepo2cwl-5flpcqqfkp-2frepo1620257096 And I can see all my files there (the custom module it is complaining about). It seems like cwltool is not copying over all the files and maybe just copying over the input file I have?

Question: if one gets their repo to run on mybinder, does it guarantee that it will work with cwltool?

To Reproduce Steps to reproduce the behavior:

  1. git clone https://github.com/iperezx/edge-plugin-smokedetect
  2. cwltool main.cwl modelValues.yml
  3. or to reproduce the main.cwl file use mic notebook read https://github.com/iperezx/edge-plugin-smokedetect
  4. See error

Expected behavior Succefully run my model with cwltool

MIC log file This comes from cwltool and not mic

➜  edge-plugin-smokedetect git:(master) ✗ cwltool main.cwl modelValues.yml                    
INFO /usr/local/bin/cwltool 3.1.20210426140515
INFO Resolved 'main.cwl' to 'file:///Users/iperezx/Documents/edge-plugin-smokedetect/main.cwl'
INFO [job main.cwl] /private/tmp/docker_tmpjzuqe2n8$ docker \
    run \
    -i \
    --mount=type=bind,source=/private/tmp/docker_tmpjzuqe2n8,target=/bmdrpN \
    --mount=type=bind,source=/private/tmp/docker_tmp7_jm2dc_,target=/tmp \
    --mount=type=bind,source=/var/folders/j6/f847737s17s4hwjc1g8n46600000gn/T/tmp_iav3um9,target=/var/lib/cwl/stg620fb507-d0ab-4d89-8586-5512f6b49bec/model.tflite,readonly \
    --workdir=/bmdrpN \
    --read-only=true \
    --net=none \
    --user=501:20 \
    --rm \
    --cidfile=/private/tmp/docker_tmpv8bnyae1/20210505164510-524413.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/bmdrpN \
    r2d-2fvar-2ffolders-2fj6-2ff847737s17s4hwjc1g8n46600000gn-2ft-2frepo2cwl-5f8e7gbs7j-2frepo1620255658 \
    /app/cwl/bin/main \
    -- \
    --cameraType \
    0 \
    --modelPath \
    /var/lib/cwl/stg620fb507-d0ab-4d89-8586-5512f6b49bec/model.tflite \
    --siteID \
    0
tee: /app/.jupyter-server-log.txt: Read-only file system
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/app/cwl/bin/main in <module>
     54     parser.add_argument('--modelPath', type=pathlib.Path, required=True)
     55     args = parser.parse_args()
---> 56     main(cameraType=args.cameraType, siteID=args.siteID, modelPath=args.
     57         modelPath)

/app/cwl/bin/main in main(cameraType, siteID, modelPath)
      9 def main(cameraType, siteID, modelPath):
     10     import numpy as np
---> 11     import inference, hpwren
     12     import tflite_runtime.interpreter as tflite
     13     import time, datetime, os, sys, subprocess

ModuleNotFoundError: No module named 'inference'
INFO [job main.cwl] Max memory used: 0MiB
ERROR [job main.cwl] Job error:
("Error collecting output for parameter 'imagePath':\nmain.cwl:25:7: Did not find output file with glob pattern: '['./hpwren-image-used-for-inference.jpeg']'", {})
WARNING [job main.cwl] completed permanentFail
{}
WARNING Final process status is permanentFail

Environment (please complete the following information):

mosoriob commented 3 years ago

Thanks for the report. I'm going to review it today.

mosoriob commented 3 years ago

I found the issue. Working on it! Thanks

mosoriob commented 3 years ago

Hi @iperezx

I have fixed the error but there is a new error. 

docker \                                   
    run \
    -i \
    --mount=type=bind,source=/tmp/s82ixmxe,target=/rRonTx \
    --mount=type=bind,source=/tmp/xtpgzf84,target=/tmp \
    --mount=type=bind,source=/tmp/tmpqs797qkh,target=/var/lib/cwl/stg3290e08d-5280-4d77-abd2-55f7b092ecc6/model.tflite \
    --workdir=/rRonTx \
    --user=10004:20163 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/rRonTx \
    --cidfile=/tmp/rylcdslw/20210511085108-376104.cid \
    r2d-2ftmp-2frepo2cwl-5ffwz76jxd-2frepo1620745894 \
    /app/cwl/bin/main \
    -- \
    --cameraType \
    0 \
    --modelPath \
    /var/lib/cwl/stg3290e08d-5280-4d77-abd2-55f7b092ecc6/model.tflite \
    --siteID \
    0
Starting smoke detection inferencing
Get image from HPWREN Camera
Image url: http://hpwren.ucsd.edu/cameras/L/tje-1-mobo-c.jpg
Description:  Unknown direction Color Original
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
/app/cwl/bin/main in <module>
     54     parser.add_argument('--modelPath', type=pathlib.Path, required=True)
     55     args = parser.parse_args()
---> 56     main(cameraType=args.cameraType, siteID=args.siteID, modelPath=args.
     57         modelPath)

/app/cwl/bin/main in main(cameraType, siteID, modelPath)
     29     testObj.urlToImage(imageURL)
     30     testObj.writeImage(imagePath)
---> 31     interpreter = tflite.Interpreter(model_path=modelPath)
     32     interpreter.allocate_tensors()
     33     result, percent = testObj.inference(interpreter)

/srv/conda/envs/notebook/lib/python3.6/site-packages/tflite_runtime/interpreter.py in __init__(self, model_path, model_content, experimental_delegates)
    202       self._interpreter = (
    203           _interpreter_wrapper.InterpreterWrapper_CreateWrapperCPPFromFile(
--> 204               model_path, self._custom_op_registerers))
    205       if not self._interpreter:
    206         raise ValueError('Failed to open {}'.format(model_path))

NotImplementedError: Wrong number or type of arguments for overloaded function 'InterpreterWrapper_CreateWrapperCPPFromFile'.
  Possible C/C++ prototypes are:
    tflite::interpreter_wrapper::InterpreterWrapper::CreateWrapperCPPFromFile(char const *,std::vector< std::string > const &,std::string *)
    tflite::interpreter_wrapper::InterpreterWrapper::tflite_interpreter_wrapper_InterpreterWrapper_CreateWrapperCPPFromFile__SWIG_1(char const *,PyObject *)
iperezx commented 3 years ago

Hi @mosoriob, that error might be coming from the value type that is being passed into the cameraType parameter. I can try to figure out if that is the case, but how did you generate the above docker command? Did you pass in different flags to cwltool? What version are you using for cwltool? I tried running it again (cwltool main.cwl modelValues.yml) and I am still seeing the same error that it can't find the inference module.

iperezx commented 3 years ago

(Correction) I do not think there is an error with the parameter cameraType and it should work. I ran a similar docker container (not generated from repo2docker) with the same exact code and got the following with no errors:

docker run sagecontinuum/plugin-smokedetect:ai-gateway-demo --siteID 0 --cameraType 0
Starting smoke detection inferencing
Get image from HPWREN Camera
Image url: http://hpwren.ucsd.edu/cameras/L/tje-1-mobo-c.jpg
Description:  Unknown direction Color Original
Perform an inference based on trained model
Fire, 65.72%

Try a different number for siteID like 1 and you should see similar output:

docker run sagecontinuum/plugin-smokedetect:ai-gateway-demo --siteID 1 --cameraType 0
Starting smoke detection inferencing
Get image from HPWREN Camera
Image url: http://hpwren.ucsd.edu/cameras/L/bm-n-mobo-c.jpg
Description: Big Black Mountain North Color Original
Perform an inference based on trainned model
Fire, 64.29%

Not sure what is happening with the model file itself.

mosoriob commented 3 years ago

I found three issues in the CWL tool and the MIC version 2.0.3 contains the solution for all

  1. The python script is generated in another location without the other files. This generates the error ---> 11 import inference, hpwren.
  2. By default, the network is disabled.
  3. The python script rewrites the type for the InputFile. From str to Pathlib.path.

I have opened the issues on the official repository and I'm going to work to submit the changes. Meanwhile, mic is using a fork with the issues fixed

Thanks a lot for your feedback.

Next steps for you

$ pip3 install -U mic
$ mic notebook read https://github.com/iperezx/edge-plugin-smokedetect

Your execution

Here are the logs of your execution

$ cwltool main.cwl values.yml                                         
INFO /var/lib/mfosorio/.local/bin/cwltool 3.0.20210319143721
INFO Resolved 'main.cwl' to 'file:///var/lib/mfosorio/repos/mic/main.cwl'
INFO [job main.cwl] /tmp/lcq2jxwh$ docker \
    run \
    -i \
    --mount=type=bind,source=/tmp/lcq2jxwh,target=/CAbMBq \
    --mount=type=bind,source=/tmp/hvyx1kmz,target=/tmp \
    --mount=type=bind,source=/tmp/tmp0de1drm2,target=/var/lib/cwl/stg79845cd1-5ed9-4aec-8929-af7898cf655d/model.tflite,readonly \
    --workdir=/CAbMBq \
    --read-only=true \
    --user=10004:20163 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/CAbMBq \
    --cidfile=/tmp/fzm9x8p3/20210512081151-218921.cid \
    r2d-2ftmp-2frepo2cwl-5fi29yaxfn-2frepo1620830162 \
    /app/cwl/bin/main \
    -- \
    --cameraType \
    0 \
    --modelPath \
    /var/lib/cwl/stg79845cd1-5ed9-4aec-8929-af7898cf655d/model.tflite \
    --siteID \
    0
tee: /app/.jupyter-server-log.txt: Read-only file system
Starting smoke detection inferencing
Get image from HPWREN Camera
Image url: http://hpwren.ucsd.edu/cameras/L/tje-1-mobo-c.jpg
Description:  Unknown direction Color Original
Perform an inference based on trainned model
Fire, 55.99%
INFO [job main.cwl] Max memory used: 70MiB
INFO [job main.cwl] completed success
{
    "imagePath": {
        "location": "file:///var/lib/mfosorio/repos/mic/hpwren-image-used-for-inference.jpeg",
        "basename": "hpwren-image-used-for-inference.jpeg",
        "class": "File",
        "checksum": "sha1$46b3ef81e111d1b889d9b2bbf765c87db2cf4c20",
        "size": 2507,
        "path": "/var/lib/mfosorio/repos/mic/hpwren-image-used-for-inference.jpeg"
    },
    "resultsPath": {
        "location": "file:///var/lib/mfosorio/repos/mic/model-inference-results.json",
        "basename": "model-inference-results.json",
        "class": "File",
        "checksum": "sha1$4e8c078100a9698d38efb73171268f33b312d818",
        "size": 283,
        "path": "/var/lib/mfosorio/repos/mic/model-inference-results.json"
    }
}
INFO Final process status is success