search

radimspetlik / SI-DDPM-FMO

[WACV 2024] SI-DDPM-FMO: Single-Image Deblurring, Trajectory and Shape Recovery of Fast Moving Objects with Denoising Diffusion Probabilistic Models
https://github.com/radimspetlik/SI-DDPM-FMO
MIT License
4 stars 2 forks source link

Synthetic Dataset Generation #3

Closed fatima-dev103 closed 1 month ago

fatima-dev103 commented 1 month ago

I am working on my master's thesis and using the open-source code provided in your repositories to generate a synthetic dataset. However I am facing some issues while developing it. In the readme.md file, it is written that the resultant dataset should be of 1TB. But when I run the bash script "run_render_parallel.sh" in renderer package of DeFMO, it only generates the dataset consisting of less than 50 objects and takes memory up to a few MBs above 200. While running the bash script, I found a lot of "Rendering failed" messages proceeded with contrast messages sometimes. I think I am missing something while dataset generation, or I have not downloaded the required datasets appropriately. I have followed the steps given in the readme.md file to download the required datasets and unpack vot 2018 using trackdat scripts. The one thing I could not come up with properly is downloading the sports 1m dataset. There are no clear steps or guidance related to the download process of this dataset. Please help me in this regard

fatima-dev103 commented 1 month ago

I have downloaded sports-1m dataset from kaggle: https://www.kaggle.com/datasets/xxcious/sports-1m. It contains one directory for test images and a numerous directories for train images. In settings.py file of renderer, the path for this dataset was like this sports-1m/seq. It is not obvious from this path declaration that how many sequences are exactly used from the sports-1m dataset, either only test or train or both.

I believe my synthetic dataset generation part should be perfect and should generate the same dataset as generated in the publications, to reproduce the results on benchmark datasets and retraining of the methods from scratch using synthetic dataset.

fatima-dev103 commented 1 month ago

image Above is a sample output logs in the terminal while I am running run_render_parallel.sh.

radimspetlik commented 1 month ago

Dear @fatima-dev103, it is common to have these errors as we check for "badly" rendered objects - objects with too small contrast, etc.

Note that we did not created the main render script, we just modified it for our purposes and you should do so too.

Take a look at https://github.com/radimspetlik/SI-DDPM-FMO/blob/217a300bf7999d8518468e3bc7c272f96aa1a3ab/renderer/render_fmo.py#L451C1-L452C59 and bellow.

A line that starts with "### YOU CAN WRITE YOUR OWN IMPLEMENTATION TO GENERATE DATA" actually should be "### YOU SHOULD WRITE..."

Take a look at the scripts and try to play around with the arguments, please. We do not intend to support render generation script issues as it is not our contribution.

Be sure to check the variable "g_number_per_category" and the file "renderer/settings.py".

Good luck...

fatima-dev103 commented 1 month ago

Dear @radimspetlik, thanks for your quick response to my questions.

Can you help me still with a few things:

1- What is the total memory of the resultant dataset you generated with the render script? I see it was 1TB as you mentioned in the readme files. I want to reproduce the results, for that, I must generate the same dataset as you generated. 2- Can you share the reference from where you get the render scripts and make changes for your purpose? 3- Can you please highlight or share the procedure you downloaded or get the sports-1m sequences? 4- Can you please mention if you have used whole sequences in the sports-1m dataset or a part of it? Can you mention highlight which part or videos you have used while dataset generation? 5- It would be great if you can also highlight the VOT dataset year, you have used while dataset generation?

radimspetlik commented 1 month ago

Hi @fatima-dev103,

  1. It is over 1TB - you cannot generate the exact same copy of the dataset as it was generated with randomness. Nevertheless, if you generate around 1TB of training data, you will get very similar results - the size of the dataset is only to ensure that the network does not overfit. From our experiments it seems that it does not really matter what kind of fast moving objects are generated as far as there is "enough" variability in the data so the network cannot overfit. I am pointing you again to the variable "g_number_per_category" - with this variable, you set the number of objects generated per category, and 1000 should be enough (or not - we try random viewpoints and random shifts in contrast/lightness/colors and then check for suitability of the result afterwards, you may need to run the script multiple times...) to generate the 1TB. Next, make sure you generate all the categories - the scripts are prepared to by run in a parallel processing environment where multiple jobs are executed at the same time. The procedure is simple, you "just generate" until you "have enough".

    start_index = int(argv[0]) % 50
step_index = int(argv[1])

    this looks like 50 jobs in parallel.

  2. my source is https://github.com/rozumden/DeFMO

  3. https://cs.stanford.edu/people/karpathy/deepvideo/ as in README - the point of the test (validation) dataset is to have a bit different data distribution than in test. It does not matter, in the end, which data exactly are used...

  4. The scripts used in their exact form are in the repo - check the scripts, please...

  5. I think that it does not matter, but it was 2016 and 2018, I believe. We also tried random backgrounds from COCO dataset and it worked too.

BTW I just updated render_fmo.py - there was a "sphere" object set for all renders, an artifact of one of our experiments. Now it should generate all objects, not just a 'sphere'.