search

satojkovic / DeepLogo

A brand logo detection system using tensorflow object detection API.
MIT License
503 stars 157 forks source link

Unable to process data in gen_train_valid_test.py script #4

Open frey123 opened 7 years ago

frey123 commented 7 years ago

Hey guys,

i have problems running the gen_train_valid_test.py script. Seems to be a shape issue while loading the pickle data.

I used the flickr27_logos dataset. The crop and augmentation script worked just fine.

I get following exception:

('Unable to process data from', 'flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/test.pickle', ':', ValueError('could not broadcast input array from shape (1387,32,64,3) into shape (1851,32,64,3)',)) Traceback (most recent call last): File "gen_train_valid_test.py", line 191, in <module> main() File "gen_train_valid_test.py", line 177, in main train_datasets, TRAIN_SIZE, VALID_SIZE) File "gen_train_valid_test.py", line 128, in merge_datasets train_dataset[start_t:end_t, :, :, :] = train_logo ValueError: could not broadcast input array from shape (1387,32,64,3) into shape (1851,32,64,3)

It processes all classes and failed when it is loading the single pickle files inside the class directories i think.

Has anyone an idea how to fix this issue?

best regards and thanks Frey

satojkovic commented 7 years ago

@frey123

Thank you for the issue report.

According to above exception message, train_datasets seem to have Ford/test.pickle incorrectly. Did you include some extra files in each class folders?

frey123 commented 7 years ago

@satojkovic

no i ddnt include extra files. It was a clean checkout and i didnt change anything. just followed your script pipeline.

skeeph commented 7 years ago

Hello, I've got similar error ('Unable to process data from', 'flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/test.pickle', ':', ValueError('could not broadcast input array from shape (1279,32,64,3) into shape (1851,32,64,3)',)) Traceback (most recent call last): File "gen_train_valid_test.py", line 190, in <module> main() File "gen_train_valid_test.py", line 176, in main train_datasets, TRAIN_SIZE, VALID_SIZE) File "gen_train_valid_test.py", line 127, in merge_datasets train_dataset[start_t:end_t, :, :, :] = train_logo ValueError: could not broadcast input array from shape (1279,32,64,3) into shape (1851,32,64,3)

satojkovic commented 7 years ago

@frey123 @skeeph

What is the output of the following command after executing the crop_and_aug.py ?

$ find flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images -name "train*" -o -name "test*"

Here is my output. Is your output same?

flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/train
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/test
flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/train
skeeph commented 7 years ago

@satojkovic Yes, but not this order. And while running crop_and_aug.py I got output train_annotation: 4536, 7 ('Skip: ', u'2662264721.jpg') ('Skip: ', u'2662264721.jpg') ('Skip: ', u'2662264721.jpg') ('Skip: ', u'2662264721.jpg') ('Skip: ', u'2662264721.jpg') original: 1079 cropped: 217488

satojkovic commented 7 years ago

@skeeph

output message is fine. But the order is important because gen_train_valid_test.py assume that test and train subfolders are alternately processed. Could you show me the output of the above find command?

mursalal commented 7 years ago

Hey, I have same problem like skeeph. Output of the find command:

flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/test

frey123 commented 7 years ago

Hey,

strange, i did a clean checkout and now it works. Only one thing changed. I'm on Debian Linux instead of CentOS now.

Best regards. Frey

skeeph commented 7 years ago

@satojkovic here is my output find flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images -name "train*" -o -name "test*" flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Texaco/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Starbucks/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Mini/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ferrari/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/HP/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Intel/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Cocacola/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Pepsi/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Sprite/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/RedBull/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Fedex/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Adidas/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/DHL/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Unicef/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Puma/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Citroen/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Yahoo/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nike/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/BMW/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Vodafone/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/McDonalds/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Google/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Nbc/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Ford/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Porsche/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Apple/test flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/train flickr_logos_27_dataset/flickr_logos_27_dataset_cropped_augmented_images/Heineken/test

satojkovic commented 7 years ago

@skeeph @mursalal

I don't know why but the order is different from my result on mac osx. What OS are you using? "test" is the first and "train" is the second in my output. "train" is the first and "test" is the second in your output. Besides my ouptut is sorted but your ouptut isn't.

Could you try the following?

gen_train_valid_test.py, at line 172,173

    #train_datasets = maybe_pickle(train_test_dirs[1::2])
    #test_datasets = maybe_pickle(train_test_dirs[0::2])
    train_datasets = maybe_pickle(train_test_dirs[0::2])  # replace 1 with 0
    test_datasets = maybe_pickle(train_test_dirs[1::2])   # replace 0 with 1
mursalal commented 7 years ago

@satojkovic

That helped. Thank you!

prakritidev commented 7 years ago

I have this problem in this script:

AttributeError: module 'common' has no attribute 'CLASS_NAME'

Can you please help me

satojkovic commented 6 years ago

@prakritidev

Could you try new master? 01146bd

asbiju2000 commented 5 years ago

On what bases need to set below values I have one class TRAIN_SIZE = VALID_SIZE = TEST_SIZE =