search

watson-developer-cloud / python-sdk

:snake: Client library to use the IBM Watson services in Python and available in pip as watson-developer-cloud
https://pypi.org/project/ibm-watson/
Apache License 2.0
1.46k stars 827 forks source link

Unable to classify zip files #200

Closed bcruga closed 7 years ago

bcruga commented 7 years ago

I've been trying to classify a .zip file using the example command:

from os.path import join, dirname
import json
from watson_developer_cloud import VisualRecognitionV3 as vr

instance = vr(vr.latest_version, api_key=["MY_API"])

with open(join(dirname(__file__), '..//file.zip'), 'rb') as image_file:
    results = instance.classify(images_file=image_file)
print(json.dumps(results))

however, the only result I'm getting is this error

WatsonException: Error: No images were specified., Code: 400

Is there a specific way to zip files for Watson's VR?

kognate commented 7 years ago

how many images are in the zip file file and how big are they? If the zip file has more than 10 images and/or the images are larger than 2mb sometimes the error returned is not helpful.

Also, what version of python are you using?

bcruga commented 7 years ago

i've tried with 10 and 5 per zip, but all of them return the same. The file size is about 18kb, very small.

I'm using python 3.5

jsstylos commented 7 years ago

Can you upload one of the zip files somewhere so we can reproduce?

bcruga commented 7 years ago

35.zip

jsstylos commented 7 years ago

This zip file works for me in Python 3.5.0 and 2.7.12. Are you sure that '..//file.zip' is picking up the right file? Do any other local files work for classification?

bcruga commented 7 years ago

if i put a picture in which is in the same folder as the .zip, it works, but every zip i try, it fails. yeah the '..//file.zip' is the path to the zip file. I even tried putting both together in the same folder, and simply using with open ('35.zip', 'rb') ... but still gives the same error.

jsstylos commented 7 years ago

One thing to try is to verify that the file is being loaded correctly by listing the contents of the zip file before trying to classify it:

with open(join(dirname(__file__), '../35.zip'), 'rb') as image_file:
    import zipfile

    zz = zipfile.ZipFile(image_file)
    zz.printdir()
    zz.close()

For me this prints out:

File Name                                             Modified             Size
35.png                                         2017-04-11 16:14:52         3268
36.png                                         2017-04-11 16:14:52         3133
37.png                                         2017-04-11 16:14:52         2828
38.png                                         2017-04-11 16:14:52         2594
39.png                                         2017-04-11 16:14:52         2661
40.png                                         2017-04-11 16:14:52         2878
bcruga commented 7 years ago

File Name Modified Size 35.png 2017-04-11 16:14:52 3268 36.png 2017-04-11 16:14:52 3133 37.png 2017-04-11 16:14:52 2828 38.png 2017-04-11 16:14:52 2594 39.png 2017-04-11 16:14:52 2661 40.png 2017-04-11 16:14:52 2878

For me as well

jsstylos commented 7 years ago

Very weird. Do you have the same issue when trying to use the Node SDK? Can you reproduce the issue in another environment?

bcruga commented 7 years ago

Sorry I can't reproduce it in another environment, only python.

jsstylos commented 7 years ago

Can you reproduce it on a different machine?

davidsilvam commented 7 years ago

Same problem here. Someone solved?

jsstylos commented 7 years ago

@davidsilvam I was never able to reproduce the original issue. Do you have a zip file that fails to process with the Python SDK but works with https://watson-api-explorer.mybluemix.net/apis/visual-recognition-v3#!/visual45recognition/post_v3_classify ? If so can you upload that file somewhere to test?

chughts commented 7 years ago

I don't think that this is Python specific, as this developerworks forum conversation ( https://developer.ibm.com/answers/questions/377690/error-400-when-trying-to-create-a-custom-classifie/ ) refers to a similar problem, but this time making use of both cURL and Node.js.

I think the problem is actually OS related, in particular the way the OS handles zip folders. I don't know which OS yet. So anyone facing this problem, can you please tell us which OS you are running on.

jsstylos commented 7 years ago

@chughts The 35.zip file above exhibits the problem for the original poster, but I can't reproduce it. If we can get a reproducible source file then we should be able to address the problem.

chughts commented 7 years ago

@jsstylos Just seen this response to my response to the question in developerworks answers -

Thanks, it works now! I have tried before placing the images at the root level inside the zip, but it didn't work so I decided to follow IBM's example. Now I decided to recreate my zips inside a Linux environment.. I'm not really sure if it has anything to do with it, but it works and I'm really thankful for your help!

jsstylos commented 7 years ago

@chughts It would be good to figure out what types of zip files don't work with the service so that we can any problems at the service level. Do you know of any zip files that can reliably reproduce the problem?

chughts commented 7 years ago

@jsstylos You won't like this. Create a folder (say for bulbs) and name it bulbs.zip then copy image files into it. I know its not a zip, but I think that is what they have been doing.

jsstylos commented 7 years ago

@chughts Do you have a file like that that reliably fails?

chughts commented 7 years ago

Ones that give me a

curl: (43) A libcurl function was given a bad argument

error

neilsbhargava commented 7 years ago

Hi @jsstylos, another user checking in having the same issue. I have a zip file that I am trying to test against a trained visual recognition classifier and I am getting the following error:

Traceback (most recent call last): File "daimlerVR.py", line 35, in <module> damaged_results = vr.classify(images_file=test_set, threshold=0.1, classifier_ids=['SimpleDamage_1258166596']) File "/Users/neilbhargava/Documents/colab/daimler-vr/env/lib/python2.7/site-packages/watson_developer_cloud/visual_recognition_v3.py", line 156, in classify params) File "/Users/neilbhargava/Documents/colab/daimler-vr/env/lib/python2.7/site-packages/watson_developer_cloud/visual_recognition_v3.py", line 133, in _image_call accept_json=True) File "/Users/neilbhargava/Documents/colab/daimler-vr/env/lib/python2.7/site-packages/watson_developer_cloud/watson_developer_cloud_service.py", line 320, in request raise WatsonException(error_message) watson_developer_cloud.watson_developer_cloud_service.WatsonException: Error: No images were specified., Code: 400

Here are the zip contents:

File Name Modified Size scanID9180_Seite-Rechts-1_00019.jpg 2017-05-02 23:26:32 1376339 scanID9200_dents_Front-Rechts-1_00002.jpg 2017-05-02 18:30:16 573645 scanID9235_corrosion_Dach-1_00005_2.jpg 2017-05-19 13:20:40 603589 scanID9265_dents_Front-Links-1_00006.jpg 2017-05-02 18:30:18 480499 scanID9396_corrosion_Seite-Links-1_00009.jpg 2017-05-02 18:30:18 408366 scanID9479_scratch_Seite-Rechts-1_00006.jpg 2017-05-02 18:30:18 629073 scanID9502_Unterboden-1_00020.jpg 2017-05-02 23:26:34 547459 scanID9668_scratch_Seite-Links-2_00019.jpg 2017-05-02 18:30:18 347631 scanID9882_dents_Dach-1_00002.jpg 2017-05-02 18:30:18 1490281 scanID9941_dents_Seite-Rechts-1_00012.jpg 2017-05-02 18:30:18 647684

I am using Mac OS X (10.11) and Python 2.7

jsstylos commented 7 years ago

@neilsbhargava Can you upload a zip file for testing?

neilsbhargava commented 7 years ago

@jsstylos - I can't upload the zip here due to NDA restrictions. I am an IBMer working in the Watson Org so if you are on Slack I can give you the zip privately (provided it is not shared with anyone outside of IBM).

jsstylos commented 7 years ago

One issue here is that some zip files are returning 413 Request Entity Too Large errors (with an html content type :-/ ), even when they don't exceed the documented file size limits. This should go away in an upcoming release, but one thing to test in the meantime is what happens when attempting to classify via curl:

curl -X POST -F "images_file=@test.zip" "https://gateway-a.watsonplatform.net/visual-recognition/api/v3/classify?api_key=your-api-key&version=2016-05-20" -v

If anyone has a file that succeeds via curl but reliably fails via the Python SDK I would very much like to see it.

If anyone has a file that fails via curl but should succeed (the files are below the stated size limits, etc.) then that would be good to see also, although do to an upcoming implementation change it is likely to get fixed in an upcoming change to the service regardless.

davidsilvam commented 7 years ago

@jsstylos I tested the upload the same .zip in https://watson-api-explorer.mybluemix.net/apis/visual-recognition-v3#!/visual45recognition/post_v3_classify in Windows 10 and in Debian. The response in Debian was positive, while in Windows the return erro is "description": "No images were specified.", "error_id": "input_error"...

chughts commented 7 years ago

Could this be related to creating zips using npm archiver - RE this response on DeveloperWorks Answers?

OK, I will try to simplified the steps for recreation tomorrow. And don't think it is a zipping problem as when I increase the image number from 15 to 40 in the zip file. it works completely fine with the same zipping method - I am actually using NPM archiver to zip the images in the folder.

chughts commented 7 years ago

Attaching 2 zip files that fail - 2 x Positive Images. fix.zip swap.zip

5agado commented 7 years ago

Any updates regarding the issue? I am encountering the same problem. Single zip file with single image using Python 3.4 on Windows 7.

germanattanasio commented 7 years ago

When sending the requests are you sending application/octet-stream as Content-Type? because the service is expecting that along with the zip file.

OndrejSzekely commented 7 years ago

I have the same problem. It is working fine on Mac OS - zip folder, and score ...On windows error from watson service - no image inside folder. Could you please fix it?

rosine commented 7 years ago

Hello, could anyone solve the problem? I still get the problem even when using a zip file containing just one image. I am using windows 10.