I'm currently working on applying your pretrained model to another environment, and I have come across a couple of questions related to your evaluation code while working with the CHI3D dataset. I'd appreciate your guidance on the following:
Train/Test Data Split
In the uploaded evaluation code, it seems that the same data object is being used regardless of whether the input key is set to train or test (as seen here). The data object is created by the get_dataset_loader function (located here), where the split parameter defaults to train. As a result, the data object contains the training set (293 samples) even though the key is set to test, which should use the test set (74 samples). Then, do I have to remake data_loader using split="test" to utilize testset correctly or is there another way?
args.num_samples parameter
I also have question regarding the num_samples parameter (in here). It appears that this parameter is intended to limit the number of samples used for evaluation. However, due to a potential issue in this code lines, this works only when num_samples < len(dataiterator.dataset). For example, when I set num_samples = 100, it properly generated 100 samples because 100 < len(dataiterator.dataset). However, when I set num_samples = 1000, it only generated 232 samples.
In your paper section 4.1, you mention testing with 1,000 samples. Should I run the generation code multiple times to collect the full 1,000 samples, or have I misunderstood something?
I'm currently working on applying your pretrained model to another environment, and I have come across a couple of questions related to your evaluation code while working with the CHI3D dataset. I'd appreciate your guidance on the following:
Train/Test Data Split In the uploaded evaluation code, it seems that the same
dataobject is being used regardless of whether the input key is set totrainortest(as seen here). Thedataobject is created by theget_dataset_loaderfunction (located here), where thesplitparameter defaults totrain. As a result, thedataobject contains the training set (293 samples) even though the key is set totest, which should use the test set (74 samples). Then, do I have to remakedata_loaderusingsplit="test"to utilize testset correctly or is there another way?args.num_samplesparameter I also have question regarding thenum_samplesparameter (in here). It appears that this parameter is intended to limit the number of samples used for evaluation. However, due to a potential issue in this code lines, this works only whennum_samples < len(dataiterator.dataset). For example, when I setnum_samples = 100, it properly generated 100 samples because100 < len(dataiterator.dataset). However, when I setnum_samples = 1000, it only generated 232 samples. In your paper section 4.1, you mention testing with 1,000 samples. Should I run the generation code multiple times to collect the full 1,000 samples, or have I misunderstood something?