Open ziyigogogo opened 6 years ago
Hi @ziyigogogo I am really happy that you noticed our work!
Quick brag/disclaimer: it turns out that with different post-processing params (erode:1 dilate:2) + TTA + score heuristics we were able to get 0.9428
precision 0.9542
recall. Without heuristics (that we want substitute with lgbm) we get 0.936
precision.
In our current solution we are not using CRF since it was very slow (do you know of any good/fast implementation). What is the boost from CRF that you are getting?
Now on the subject of using lgbm, our procedure is:
Currently lgbm step is done with heuristics but we should have it very soon.
What is important is that our approach doesn't (unfortunately) deal with the round vs rectangle problem. That being said my idea to adjust our approach to deal with that would be the following:
I will get back to this as soon as we have lgbm implementation open for you to see and discuss further.
Hi,@jakubczakon,
Our team had a little break for the past 3 days, sorry for the late response.
Thanks a lot for your detail procedure and deep thought for my question, which is absolutely an inspiration for me. I will discuss this post with my teammates today and maybe do some quick implementation based on this. Once we draw any progress, i will post back.
For your question "Do you know of any good/fast implementation".
Since I saw your CRF related code here line284-302. I believe we are using the same package(pydensecrf) which maybe the only open-source package available now. So the answer is no, we currently dont have a faster crf implementation. Recently a paper introduced a conv method for CRF--the authors claim a 100x speed-up . However we did not try it so far, since it seems that the boost from end-to-end trainable CRF network is limited and changing the entire network architecture for such a little improvement is not worth for now.
Another question "What is the boost from CRF that you are getting". Actually i have described this above, maybe you lost me due to my poor standard of English, sorry : )
Let me directly use an
to explain, as you can see, for this very example, by applying CRF to our network prediction(white), we got a more "rectangle-like" mask(red). But still a lot to go because the quality of CRF post-processing varies a lot as i mentioned(it only works good on a part of our test set). That's why our team wants to utilize some tree-like networks(maybe lgbm) helping us pick the best result from multiple techniques(like a combination of crf-enabled output, mask-rcnn output and unet output).
Finally, looking forward for your up-coming release. Thanks a lot!
Regards
@ziyigogogo merged lgbm (and random forest) second level models to master with #152
Hi there,
I am really thankful for your sharing at CrowdAI.
I myself is now in the industry of Satellite imagery. And our team has studied the ML solution of this for more than a year now. Our unet solution for our clients can also reach 0.93+ (IoU >= 0.5) result at present with some morphology post-processing (not for this CROWDAI competiton, but for our own business project with our own datasets).
And so far, we are exploring a way to improve the quality of the outputs. Say, one of the most significant difference between masks produced by CNN and Man-made labels is the "straightness". For example, CNN masks usually have round corners(or curve contour) while man-made labels usually are standard rectangles(90-degree corner and straight outlines).
We are now searching for solution for this problem. Currently, two main directions are post-processing and output ensembling. Same as yours, we also introduced CRF post-processing in our solution, however, the quality of the CRF-enabled outputs varies a lot. Some of them are extremely straight and perfectly match our GT, while others are not that good.
So, we recently started a research about result ensemble tech. One of our current idea is that we use LightGBM on top of several of our different model outputs and CRF-enabled outputs. Since while "CRF-enabled" results have more noise and error-classification on pixels, it also represent a more "stragiht" style on the masks. So maybe, the ensemble of outputs from CRF and other models can help.
As i noticed that you guys mentioned "lightgbm-based" post-processing, actually we want hear more about your ideas about how to utilize lightgbm on this problem so that maybe we could work together to solve this problem?
Regards Ziyi