how to create dataset

[Niculuse]

Hi,I'm a new learner.I do not understand how to prepare dataset for train,eventhough I've read your instruction.I download market1501 dataset,and I create a folder named images to include all imgaes of dataset.Well,there are so many pictures in dataset,I don't understand how to create a new .csv file to describe dataset.I cloned all your project files in a local folder where I see .csv files in data folder.Thus,I pass the step.I got a problem as follows: "PS F:\tf\ReID> python train.py --train_set data/market1501_train.csv --image_root images --experiment_root experiments/my_experiment Training using the following parameters: batch_k: 4 batch_p: 32 checkpoint_frequency: 1000 crop_augment: False decay_start_iteration: 15000 detailed_logs: False embedding_dim: 128 experiment_root: experiments/my_experiment flip_augment: False head_name: fc1024 image_root: images initial_checkpoint: None learning_rate: 0.0003 loading_threads: 8 loss: batch_hard margin: soft metric: euclidean model_name: resnet_v1_50 net_input_height: 256 net_input_width: 128 pre_crop_height: 288 pre_crop_width: 144 resume: False train_iterations: 25000 train_set: data/market1501_train.csv Traceback (most recent call last): File "train.py", line 427, in main() File "train.py", line 226, in main pids, fids = common.load_dataset(args.train_set, args.image_root) File "F:\tf\ReID\common.py", line 132, in load_dataset csv_file, image_root, missing_count, len(fids))) OSError: Using the data/market1501_train.csv file and images as an image root 12936/12936 images are missing" Please tell me how to do.Thanks for your time and your reply!

[lucasb-eyer]

The image_root needs to be the full path to the image folder. If your image folder is not in the same folder as the code, then your invocation doesn't work.

If your image folder is, for example, in D:\data\market\images, then you need to pass that as --image_root D:\data\market\images. The rule is that when concatenating the image_root and the names that are in the .csv file, they should form a full path to the image files, for example D:\data\market\images\query/0001_c1s1_001051_00.jpg.

Note that the images folder does need to have the sub-folders query, bounding_box_test, and bounding_box_train, as in the original dataset; you can see that by looking at the csv file.

[mayidu]

hello lucasb-eyer, I don't understand how to create .csv file, there are many images, that need we write the full path in to .csv, the work will need much time. I have a other question that the code can train the cars to Re-ID?

[lucasb-eyer]

Well of course you shouldn't do this by hand. Either use standard unix commands and piping/redirection, or write a small program in your favourite language to do so. Explaining that in more detail is outside of the scope of support I'm willing to give here.

Yes, it can train cars to re-id, I have used it for that in the past.

[mayidu]

Thanks you reply, can you tell me the data set you used train cars?

[lucasb-eyer]

Sure, I tested it on CARS196, but because the dataset is quite small, you need to train for a very short time (just a few thousand updates) or it will overfit.

[mayidu]

Thanks a lot, I give it a try！

[mayidu]

I download the dataset, whether the dataset need to arrange for training. I think the format should be that one car have many images from different angles.

[lucasb-eyer]

The dataset is already labeled, one class equals one identity (PID), so just convert it to our .csv format. Unless of course you want to do something else, but that's up to you to figure out for your specific task, I wouldn't know.

[mayidu]

Thank you very much. I have a question about Tensorflow to ask for advice. Now I use this code train a model, and I got four files finally you know, which are checkpoint, .data, .meta, and .index. I want to use the trained model in deep sort for tracking, but the deep sort need .ckpt file, and he use TF-slim to restore model. I don't know how to load the trained model of Triplets Re-id to deep sort, can you give me some suggest? I am a new learner, and I need some help! Thanks!

[Pandoro]

Dear @mayidu,

these questions are not related to our project, so this is not the right place for these questions. If you actually read our code you will see how we do this. But furthermore there is the TensorFlow documentation and tons of other resources such as Stack Overflow for this kind of issues.

[mayidu]

Ok, very thanks for you to reply this question. Now I know how to do and the answer is in your code, Thank you very much again @lucasb-eyer and @Pandoro .

[mayidu]

The code only reads images from dataset and .csv, if I want to use the frame of video, How can I do? I use the embed.py code:

net_input_size = (args.net_input_height, args.net_input_width) pre_crop_size = (args.pre_crop_height, args.pre_crop_width)

# Setup a tf Dataset containing all images.
dataset = tf.data.Dataset.from_tensor_slices(data_fids)
print(dataset)

# Convert filenames to actual image tensors.
dataset = dataset.map(
    lambda fid: common.fid_to_image(
        fid, 'dummy', image_root=args.image_root,
        image_size=pre_crop_size if args.crop_augment else net_input_size),
    num_parallel_calls=args.loading_threads)

print (dataset)

# Augment the data if specified by the arguments.
# `modifiers` is a list of strings that keeps track of which augmentations
# have been applied, so that a human can understand it later on.
modifiers = ['original']
if args.flip_augment:
    dataset = dataset.map(flip_augment)
    dataset = dataset.apply(tf.contrib.data.unbatch())
    modifiers = [o + m for m in ['', '_flip'] for o in modifiers]

if args.crop_augment == 'center':
    dataset = dataset.map(lambda im, fid, pid:
        (five_crops(im, net_input_size)[0], fid, pid))
    modifiers = [o + '_center' for o in modifiers]
elif args.crop_augment == 'five':
    dataset = dataset.map(lambda im, fid, pid:
        (tf.stack(five_crops(im, net_input_size)), [fid]*5, [pid]*5))
    dataset = dataset.apply(tf.contrib.data.unbatch())
    modifiers = [o + m for o in modifiers for m in [
        '_center', '_top_left', '_top_right', '_bottom_left', '_bottom_right']]
elif args.crop_augment == 'avgpool':
    modifiers = [o + '_avgpool' for o in modifiers]
else:
    modifiers = [o + '_resize' for o in modifiers]

# Group it back into PK batches.
dataset = dataset.batch(args.batch_size)
#print (dataset)

# Overlap producing and consuming.
dataset = dataset.prefetch(1)
print (dataset)

images, _, _ = dataset.make_one_shot_iterator().get_next()

print (images)

# Create the model and an embedding head.
model = import_module('nets.' + args.model_name)
head = import_module('heads.' + args.head_name)

endpoints, body_prefix = model.endpoints(images, is_training=False)
with tf.name_scope('head'):
    endpoints = head.head(endpoints, args.embedding_dim, is_training=False)

with h5py.File(args.filename, 'w') as f_out, tf.Session() as sess:
    # Initialize the network/load the checkpoint.
    if args.checkpoint is None:
        checkpoint = tf.train.latest_checkpoint(args.experiment_root)
    else:
        checkpoint = os.path.join(args.experiment_root, args.checkpoint)
    if not args.quiet:
        print('Restoring from checkpoint: {}'.format(checkpoint))
    tf.train.Saver().restore(sess, checkpoint)

but I get the error :ValueError: Input must be of size [batch, height, width, 3], the image I got is Tensor("IteratorGetNext:0", shape=(?, 480, 3), dtype=uint8). where I error?

[lucasb-eyer]

You will have to modify the code to your needs, or write your own from scratch, taking inspiration from ours. This is really way, way beyond the support we can give.

[asgq123]

If I use the car datasets, how should I evaluate the MAP( how to build query.csv in CARS196)thanks a lot