AttributeError: 'generator' object has no attribute 'shape'

[anorlo]

Hi, I am using Tensorflow 2.4.

When I try to use my own Sequence generator, it shows

File "train_hybrid_Memory.py", line 343, in <module>
    train_and_predict(args)
  File "train_hybrid_Memory.py", line 260, in train_and_predict
    use_multiprocessing=True)
  File "/home/ziyuan/jiaming/python36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1896, in fit_generator
    initial_epoch=initial_epoch)
  File "/home/ziyuan/jiaming/python36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1099, in fit
    steps_per_execution=self._steps_per_execution)
  File "/home/ziyuan/jiaming/python36/lib/python3.6/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 1135, in __init__
    model=model)
  File "/home/ziyuan/jiaming/python36/lib/python3.6/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 909, in __init__
    **kwargs)
  File "/home/ziyuan/jiaming/python36/lib/python3.6/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 789, in __init__
    self._first_batch_size = int(nest.flatten(peek)[0].shape[0])
AttributeError: 'generator' object has no attribute 'shape'

The corresponding parts of code are:

class SequenceData(Sequence):

    def __init__(self, batch_size):
        self.batch_size = batch_size
        print("init...")

    def __len__(self):
        return math.ceil(131/self.batch_size)

    def __getitem__(self, idx):
        print(idx)
        img, _ = load(args.data + '/myTrainingData/volume-' + str(idx) + '.nii')
        tumor, _ = load(args.data + '/myTrainingData/segmentation-' + str(idx) + '.nii')
        img_list=img
        tumor_list=tumor

        maxmin = np.loadtxt(args.data + '/myTrainingDataTxt/LiverBox/box_' + str(idx) + '.txt', delimiter=' ')
        minindex = maxmin[0:3]
        maxindex = maxmin[3:6]
        minindex = np.array(minindex, dtype='int')
        maxindex = np.array(maxindex, dtype='int')
        minindex[0] = max(minindex[0] - 3, 0)
        minindex[1] = max(minindex[1] - 3, 0)
        minindex[2] = max(minindex[2] - 3, 0)
        maxindex[0] = min(img.shape[0], maxindex[0] + 3)
        maxindex[1] = min(img.shape[1], maxindex[1] + 3)
        maxindex[2] = min(img.shape[2], maxindex[2] + 3)
        minindex_list = minindex
        maxindex_list = maxindex

        f1 = open(args.data + '/myTrainingDataTxt/TumorPixels/tumor_' + str(idx) + '.txt', 'r')
        tumorline = f1.readlines()
        tumorlines = tumorline
        tumoridx = len(tumorline)
        f1.close()

        f2 = open(args.data + '/myTrainingDataTxt/LiverPixels/liver_' + str(idx) + '.txt', 'r')
        liverline = f2.readlines()
        liverlines = liverline
        liveridx = len(liverline)
        f2.close()

        # yield generate_arrays_from_file(args.b, trainidx, img_list, tumor_list, tumorlines, liverlines,
        #                                           tumoridx, liveridx, minindex_list, maxindex_list)

        while 1:
            X = np.zeros((batch_size, args.input_size, args.input_size, args.input_cols, 1), dtype='float32')
            Y = np.zeros((batch_size, args.input_size, args.input_size, args.input_cols, 1), dtype='int16')
            Parameter_List = []
            for i in range(batch_size):
                img = img_list
                tumor = tumor_list
                minindex = minindex_list
                maxindex = maxindex_list
                num = np.random.randint(0, 6)
                if num < 3:
                    lines = liverlines
                    numid = liveridx
                else:
                    lines = tumorlines
                    numid = tumoridx
                Parameter_List.append([img, tumor, lines, numid, minindex, maxindex])
            pool = ThreadPool(thread_num)
            result_list = pool.map(load_seq_crop_data_masktumor_try, Parameter_List)
            pool.close()
            pool.join()
            for i in range(len(result_list)):
                X[i, :, :, :, 0] = result_list[i][0]
                Y[i, :, :, :, 0] = result_list[i][1]
            if np.sum(Y == 0) == 0:
                continue
            if np.sum(Y == 1) == 0:
                continue
            if np.sum(Y == 2) == 0:
                continue
            yield (X, Y)

Call as :

seq = SequenceData(args.b)
    model.fit_generator(seq,
                        steps_per_epoch=steps,
                        epochs=36, verbose=1, callbacks=[model_checkpoint], max_queue_size=10, workers=3,
                        use_multiprocessing=True)

I have tried both from tensorflow.python.keras.utils.data_utils import Sequence and from keras.utils import Sequence all useless. It is very important to me. Thank you so much!

[BahramKBaloch]

@anorlo any luck in finding the solution?

[sushreebarsa]

@anorlo In order to expedite the trouble-shooting process, please provide a code snippet to reproduce the issue reported here. Please have a look at the issue for reference. Please try to execute your code using TF v2.6.0 and let us know if the issue persists ?Thanks!

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.