How to use for dataset iterator

[ramdhan1989]

Hi, I have tensorflow dataset (ds_train,ds_test), ds_info = tfds.load('grains', split=["train","val"], with_info=True, as_supervised=True) where ds_train consist of batch of image and label. how to modify in order to run in my own dataset ?

thank you

[ozora-ogino]

Hi @ramdhan1989 ,

Please check my example code. https://github.com/ozora-ogino/asoftmax-tf/blob/main/train.py

If you got errors, share with me please.

[ramdhan1989]

this is the error : `AssertionError: in user code:

C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\training.py:806 train_function  *
    return step_function(self, iterator)
C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\training.py:796 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\distribute\distribute_lib.py:1211 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\distribute\distribute_lib.py:2585 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\distribute\distribute_lib.py:2945 _call_for_each_replica
    return fn(*args, **kwargs)
C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\training.py:789 run_step  **
    outputs = model.train_step(data)
C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\training.py:747 train_step
    y_pred = self(x, training=True)
C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\base_layer.py:985 __call__
    outputs = call_fn(inputs, *args, **kwargs)
C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\functional.py:386 call
    inputs, training=training, mask=mask)
C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\functional.py:517 _run_internal_graph
    assert x_id in tensor_dict, 'Could not compute output ' + str(x)

AssertionError: Could not compute output Tensor("arc_face_2/Softmax:0", shape=(None, 28), dtype=float32)

` I suspect there is error in input format. I am struggle to split tfds into image and label to follow your format data. Is there any idea ?

thanks

[ramdhan1989]

it is working now for training using custom dataset generator. I have question, when doing training (.fit), I try to use validation data but the validation accuracy during the training is far away below training accuracy after following modification in this repo to allow prediction without label. Previously, the accuracy between training and validation just right. I wonder is there any problem during prediction method ? in your example , there is no validation data used.

please advise thanks

[ozora-ogino]

I'm not sure, so let me try.

[ramdhan1989]

I am using data for image classification. for the code I am using this github https://github.com/Hayashi-Yudai/ML_models/tree/master/ArcFace to implement arcface model and tried to modify refer to your code for prediction. Originally, I got similar accuracy between train and val data but then after changed to mimic your code the val accuracy fall far away below training accuracy.

[ozora-ogino]

I'm busy in this month and will try to fix the issue next month. I'll do my best though. If you can find and fix issues, it's really helpful for me!