No connecting objects annotations for some images

[bernhardschaefer]

First of all: Great work, it's really nice to see more people working on BPMN recognition. :-)

I wanted to try out my Sketch2BPMN approach on your dataset. When I looked at the dataset, I found out that there are much fewer images in the "BPMN Connecting Objects" dataset:	type	test	train	total
BPMN Connecting Objects	148	344	492
BPMN Nodes	254	574	828

I then realized that the images that are not part of RePROSitory do not contain connecting objects. However, there are many other images that do contain connecting objects, but these are not annotated:

0_1596633100391_2529131626102618.png, 0_1596634931897_3113029923189011.png, 10_1552924491944_343421227133030.png, 10_1552924503407_1124201624283332.png, 10_1585149129596_143011343133178.png, 10_1596633099648_9211322717642332.png, 10_1596634929223_302814272451341.png, 11_1552924487304_428102120112024.png, 11_1585149746236_301732218120122.png, 11_1596632032556_1319165724182424.png, 11_1604787484038_2722910156222814.png, 11_1648594145286_311514301335710.png, 11_1648633083664_4281822515211218.png, 12_1585149697313_2542291022291315.png, 12_1596634928934_218922111432330.png, 12_1596634934827_1121916214141232.png, 12_1617355149204_3316152034333010.png, 13_1552924496673_3013233329212233.png, 13_1596633098565_241230362510415.png, 13_1596634925633_2420031259192528.png, 13_1596634929104_289202103411205.png, 13_1596634932725_0244091731283022.png, 13_1604790818666_147131926314263.png, 14_1552924489592_2610281443216724.png, 14_1596634266346_3210194251392012.png, 14_1617353863282_192626433211256.png, 15_1552924490488_1813312526301325.png, 15_1552924494918_2143132313212010.png, 15_1596633098783_172018252251415.png, 15_1596634265992_1620253391322027.png, 15_1596635806284_6192528283141934.png, 16_1552924487378_221291382333198.png, 16_1648594145672_193128281218224.png, 17_1552924488109_513323081091614.png, 17_1552924494385_822819263162121.png, 17_1596633340753_302381112116307.png, 17_1596634928198_278711111917431.png, 17_1596634936484_225131325113132.png, 17_1648628584449_471114335151311.png, 18_1596635807060_4193120193428121.png, 19_1596634924902_181643233441116.png, 19_1604769528558_102661742312719.png, 19_1648593483136_1839102924312910.png, 1_1552924493993_1111231218102221.png, 1_1596634935664_9213083161011477.png, 1_1634658323762_41111203210121631.png, 20_1596634934982_271626728193333.png, 21_1596632660606_022507192214234.png, 21_1596634932543_453362523311111.png, 21_1648593482301_832919225202269.png, 22_1552924492330_027131603412321.png, 22_1596634931955_3213141811183222.png, 22_1604788161633_929127112920275.png, 22_1648630591777_144111920491810.png, 23_1552924487854_463114153426331.png, 23_1552924492086_3231812371025611.png, 23_1596631717364_262816718251829.png, 23_1596634268838_192210631125624.png, 23_1596635806127_122181924818220.png, 24_1552924500764_425164153152522.png, 24_1596633099416_1729142231172920.png, 24_1596633882239_321722323291725.png, 25_1596631717259_3422292814152134.png, 25_1648594708342_132133029133210.png, 27_1596635985733_341246164236211.png, 27_1648594708127_103111121656306.png, 28_1552924491482_132531131312615.png, 28_1585149240921_193434410171714.png, 28_1596632662068_615222817191825.png, 28_1634656890884_523322326313035.png, 29_1648592677836_282213311504386.png, 2_1552924487093_91522172119431615.png, 2_1596634926989_10108318205192633.png, 2_1596634932653_51523633201429310.png, 2_1604769383374_9341483424203313.png, 2_1648592679218_5191251022881525.png, 2_1648594144324_4182627192716144.png, 30_1552924496047_201629132931288.png, 30_1552924496970_5321033211241615.png, 30_1596631717478_513029103103314.png, 30_1596634266927_242517201492734.png, 31_1552924491253_2401329229122512.png, 31_1596634932143_322671430241816.png, 31_1596635807145_292601924641931.png, 32_1596632658854_3218304301573223.png, 32_1596633100161_291182992521113.png, 32_1596634928648_1042422303401422.png, 34_1596632662221_129210413311810.png, 34_1596634931531_1626102031151518.png, 34_1596634935820_121650018242326.png, 34_1604768804646_1622310422221727.png, 35_1552924497643_3341818525103431.png, 35_1648594144803_628129013730929.png, 36_1552924492667_281231102551923.png, 36_1552924495203_321320010733122.png, 36_1552924502457_8176212428171021.png, 36_1648592678084_2705152717229626.png, 36_1648594143679_213342195023314.png, 37_1552924503947_2313924135922632.png, 37_1596633340213_8722341331221113.png, 37_1596634937730_332452411073017.png, 37_1596634937885_342813342127213.png, 37_1596635987411_2227132142312831.png, 38_1552924491306_1311137238146017.png, 38_1552924498543_0118028335232312.png, 38_1596634935366_3164213112818124.png, 38_1596635986435_2933191681836632.png, 39_1552924486947_1227162843142721.png, 39_1596635986089_693310131417012.png, 39_1648594143780_251582571319619.png, 39_1648594145962_012342418212019.png, 3_1552924497499_61418202614151013.png, 3_1596634267504_2924121201583318.png, 3_1648592677582_1923311102322293.png, 40_1552924489705_163419969253422.png, 40_1552924497031_101917522133818.png, 40_1596634934508_142178238311630.png, 40_1596635986218_1251591418331223.png, 41_1552924481946_261671832519268.png, 42_1552924493661_232911211422113.png, 42_1596634927465_2016141331243414.png, 42_1596634937361_1829525276501220.png, 43_1596633099311_173287251314620.png, 43_1596634267304_633413351732516.png, 43_1596634924683_2871302333231433.png, 43_1596634927824_309428223121310.png, 43_1648594145819_152131159292220.png, 44_1552924494287_427253312941722.png, 45_1552924498484_106332923114330.png, 45_1585149015104_141311112410514.png, 45_1585149616773_713292031283229.png, 45_1592863337695_829113618273130.png, 45_1596633340020_818212016341125.png, 45_1596633882117_3117213030242728.png, 45_1596634267042_125293203117427.png, 45_1596634929419_2531217241010332.png, 46_1552924497104_1019162202331112.png, 46_1552924498780_112653233311732.png, 46_1596635986904_291672417203224.png, 46_1648592678500_2715145173120114.png, 46_1648592678948_2625173429217517.png, 47_1596632032797_011197253433301.png, 47_1596632659598_2426331123233133.png, 47_1596632660914_2311121393271122.png, 47_1596634266102_2911816416305925.png, 47_1596634933531_151581381351624.png, 47_1604740365505_192182316292722.png, 47_1648593483484_261113232210337.png, 48_1585148887575_2933266123333821.png, 49_1552924491359_181522219302169.png, 49_1585148444063_141151112913185.png, 49_1596634926322_851113151617156.png, 49_1604787587618_221714411231816.png, 4_1596631717157_25152933523231213.png, 4_1623314720193_3030192128271821.png, 5_1585149388006_2926161614161721.png, 6_1552924493092_15620213013203215.png, 6_1596634267366_0101021193451328.png, 6_1596634930085_7272813262429424.png, 7_1552924491885_2816170192620932.png, 7_1552924493511_1331214163363428.png, 7_1589720341650_1781242843291319.png, 7_1596632661187_1282326245533256.png, 7_1596634931803_8611433235219220.png, 7_1648594144095_1593253690241934.png, 8_1552924494511_14127120251127234.png, 8_1596632659303_3292417017178103.png, 8_1596634266211_1214829191728216.png, 8_1596634930007_83113150299141134.png, 9_1596634929889_17153425152516033.png, 9_1596635987232_3008321824142222.png

Is there a reason for this? Or am I missing something?

Sketch2BPMN uses one model to recognize both nodes and connecting objects. Therefore, I wanted to combine the two separated coco datasets into one, but that's not possible if only some images have edge annotations.

[FabrizioFornari]

For this version of the tool we used two models: one for recognising nodes and another for recognising connecting objects. The second one is trained on a reduced set of images due to labeling issues.

We labelled connecting objects with bounding boxes. However some connecting objects have a particular layout e.g. they may lead to the design of boxes that include several other elements (e.g. nodes). Including those images in our experiments reduced the performance of the model. Currently we are working to improve the detection of connecting objects solving this issue.

If you want to combine the datasets into a unique one you may combine those annotations that refer to an image that is in both datasets, by doing so you will exclude both the filtered images and the augmented ones (those that have no connecting objects).

[bernhardschaefer]

Thanks for clarifying this Fabrizio. Do you have the AP numbers when training the node model on the subset of the images? Or alternatively, do you have the coco files and results when training the connecting objects model on the full dataset? Otherwise I cannot do a fair comparison.

One more thing: In the coco files I see "Complex Gateway" as a category, but this category is missing in the results in the README. Did you maybe forget to report the results for this category?

[bernhardschaefer]

To reduce complexity I also went for a separated approach where I train and evaluate two models, one for recognising nodes and another for recognising connecting objects.

There are two questions that remain from my side: 1) AP-Keypoints evaluation: how did you set the kpt_oks_sigmas when creating the COCOEvaluator? The kpt_oks_sigmas is missing in the BPMN_Keypoints_Detection.ipynb notebook, and it's a required parameter in COCOEvaluator when evaluating keypoints.

2) Complex Gateway: Which bounding box AP did you get for this class?

Thanks in advance!

[FabrizioFornari]

We do not have the AP numbers for the node model on the subset of the images. As well, we discarded the coco files and results for the connecting objects model on the full dataset. However, we could obtain them. Let's discuss it and see which solution is the best.

Since Complex Gateway were not detected properly we preferred to omit them in this first prototype. However we have solution for that, but it will need us to re-train the model.

Referring to AP-Keypoints evaluation, we set the kpt_oks_sigmas to 1.0. We just noticed that the file on github is not the most updated one, we are going to fix that soon. Thanks for noticing that.

[bernhardschaefer]

Thank you for the clarification. As I also now train and evaluate two models, I can work with the coco files that are currently available.

I would appreciate if you could update the Jupyter notebook with the training configuration that you used to obtain the results in the README.