Use custom dataset

[ttsesm]

Hi @SeongJong-Yoo,

Thanks for sharing your work. It seems quite interesting. I would like to try your code with some custom datasets. Thus, I would appreciate if you could provide some further details how a custom dataset should be modified, e.g. the format of the input data, folder structure, etc. For example I would be interested to check the performance of your approach using the the 3D fragments from here and here.

Thanks.

[SeongJong-Yoo]

Thank you for having an interest in our work. Unfortunately, our pipeline is only able to apply axial symmetric objects, such as potteries, and the datasets you referred does not seem to apply in this case. If you have more questions then feel free to tell me! Thanks.

[ttsesm]

Thanks @SeongJong-Yoo for the prompt response. Yes skipping through the paper, I realized that in the current state your algorithm will work with only axial symmetric objects. Do you think though, that possibly it would work with this explicit 3D model which has such characteristics?

In any case it would be helpful to describe how a custom dataset should be given as input to your pipeline. In the data_path.h file I see that for each pot you load different parts e.g. meshes, surfaces, axes, breaklines, etc... Does someone need to have all these information extracted beforehand or it can be done on the fly during the code runtime. Because also in the description insider the paper you mention that you extract these features but it is not specified whether it is done on the fly or in advance. If I have overlooked it, please accept my apologies.

Thanks.

[SeongJong-Yoo]

Basically, in our work axial symmetric objects are the most important precondition, so that we utilize this characteristic to overall modules, such as ‘Feature extraction module’ and Refine or alignment module’. Therefore, the pipeline must be modified when applied to general objects.

Second, ‘Feature extraction module’ is not included in this code. Therefore, datasets should be prepared in a specific format. For example, the edge line is counterclockwise sorted ‘pcd’ format 3D point cloud including normal data, also axis data is required in format of ‘xyz’. For more details about data format, you can download and check our dataset.

Thanks!

[ttsesm]

Yes, I've seen the folder structure in your dataset (I've already downloaded and played a bit with the code and the output) that's why I was asking how you create this format. So if I understand it correctly this "Feature extraction" module creates these additional data folders e.g. breaklines, axis data, etc. that are needed.

Are you planning to make this "Feature extraction" module available as well in the near future or not really? I think it will be useful as I mentioned earlier possibly for future users to evaluate your pipeline with their own custom datasets.

Thanks.

[jhh37]

Dear ttsesm, Thanks for trying out our code.

We're trying to put together the data preprocessing bit and release it in the future for users to play around with their custom dataset but it may take some time and probably not in near future (e.g. not in a month's time but probably sometime next year).

Our current implementation is divided into two parts (written by different authors), axis estimation using MATLAB and surface segmentation/extraction parts in C++. These were carried out separately (with accumulated runtimes reported in the paper), and it would require some time to port everything to a single program (because axis estimation requires some optimization procedures not handled by Ceres-solver).

Another related issue is that we have not obtained explicit permission from the relevant institution regarding release of raw scans of potteries. This is why we mentioned we would release our reassembly code and preprocessed dataset in the paper.

But I agree the preprocessing part would be necessary for others to play around with their own datasets, find issues and improve the stages like in SfM. This is definitely on our to-do list, and we are currently working on it along with our extended work for a journal. Will update you on this thread once it's ready.

[ttsesm]

I see, understandable. I am looking forward to any updates :-).

In any case, thank you for your time.

[ttsesm]

Hi @jhh37, @SeongJong-Yoo

One more question, is it possible to provide the ground truth point clouds/meshes of the potteries that you provide in the dataset?

What I mean is that, if I load the individual pieces of each pot from the "Mesh" or "Surfaces" folders these are randomly positioned:

Thus, would be possible to provide them in a way where when you load each piece individually you get the actual pot.

Thanks.

[ttsesm]

Actually, in the paper it is mentioned that the counting success is extracted based on the the (semi) ground truth transformation matrices for each pot and the ones that you extract during the pipeline runtime. However, in the code is not clear where you do that comparison and moreover I do not see anywhere in the provided data these ground truth transformations. Would you please elaborate a bit on that.

Thanks.

[SeongJong-Yoo]

Hi @ttsesm,

Thank you for your continuous interest in our works. To answer your questions about ground truth transformation matrices and the comparisons between the results and GT matrices, we separately ran the codes as we mentioned before. Now we are extending our works to a Journal paper, and we are preparing to package all our codes and data including what you mentioned. So please wait a little bit more, and we hope to publish it as soon as possible. On the other hand, Can I ask specifically what you want to do with our data and code? If we know about your goal, we might help you more precisely.

Thanks.

[ttsesm]

Hi @ttsesm,

Thank you for your continuous interest in our works. To answer your questions about ground truth transformation matrices and the comparisons between the results and GT matrices, we separately ran the codes as we mentioned before. Now we are extending our works to a Journal paper, and we are preparing to package all our codes and data including what you mentioned. So please wait a little bit more, and we hope to publish it as soon as possible.

Hi @SeongJong-Yoo, thank you for the reply. I am looking forward to the journal publication as well as the release of the extra modules and GT data. I think providing the GT transformation matrices will be useful.

Another question though, it is not clear to me how you extract the Score value during the graph building procedure. Initially I thought that this one is the comparison between the GT transformation matrices and the reconstructed ones but apparently it is not. Thus, I would appreciate if you could elaborate a bit to what this value corresponds to. For example for the first pot (Pot_A) for the first of the three reconstructions I am getting a Score of 280 and so on.

----- Finish build graph  -----
Time after data load : 1358390sec
Result : 1/3, Score : 280
Logpath: /home/ttsesm/Data/repair_dataset/structure_from_sherds/Log/Log
Nodes
1->3->5->6->7->4->2->8
Edges : 14
(3-1), (5-3), (6-1), (6-3), (7-1), (7-6), (4-1), (6-4), (7-4), (2-1), (4-2), (5-2), (8-2), (8-4), 
Score : 280
Result : 2/3, Score : 254
Logpath: /home/ttsesm/Data/repair_dataset/structure_from_sherds/Log/Log
Nodes
1->3->5->6->4->2->7->8
Edges : 11
(3-1), (5-3), (6-1), (6-3), (4-1), (6-4), (2-1), (4-2), (5-2), (7-2), (8-4), 
Score : 254
Result : 3/3, Score : 213
Logpath: /home/ttsesm/Data/repair_dataset/structure_from_sherds/Log/Log
Nodes
1->2->3->5->4->7->8
Edges : 7
(2-1), (3-1), (5-3), (4-1), (7-1), (7-4), (8-3), 
Score : 213

On the other hand, Can I ask specifically what you want to do with our data and code? If we know about your goal, we might help you more precisely.

Thanks.

At the moment, I am just trying to collect all possible available 3d fragment datasets (which it seems not be many) and experiment with the SOTA (and the code that each provides if they do). For example, I have the ones from the Presious project as well as the one from the Tuwien (if you know any more that I am not aware, feel free to mention them). In principle later on we plan also to create a more extensive dataset on our own as well as to create a more generic re-assembly pipeline which would be able to address all different fragmented objects and not to focus only on potteries or statues, etc...

Thanks.

[SeongJong-Yoo]

Hi @ttsesm, The 'Score' is to generate the priority list and is the same as the 'inliers' in chapter 5.2 of our main paper. The higher the score, the better the reconstruction result. We also tried to find and gather other potteries data but we were not able to find open data set. That's one of the main reasons that we decided to share our data set and code. I hope our paper and data set help your research. If you have any other questions, please feel free to ask.

Thanks.

[ttsesm]

Hi @ttsesm, The 'Score' is to generate the priority list and is the same as the 'inliers' in chapter 5.2 of our main paper. The higher the score, the better the reconstruction result.

I see, so the actual metric for matching the different sherds is this score and not the comparison with the gt transformations of the sherds.

We also tried to find and gather other potteries data but we were not able to find open data set. That's one of the main reasons that we decided to share our data set and code. I hope our paper and data set help your research. If you have any other questions, please feel free to ask.

Thanks.

Yes, indeed it seems that there is a kind of lack in such datasets. Definitely is gonna be helpful, and I guess not only for me.