search

hammerlab / SmartCount

Repository for collaboration on Celldom computer vision solutions
Apache License 2.0
2 stars 0 forks source link

Validate M-Ladder chip configs #69

Closed eric-czech closed 5 years ago

eric-czech commented 5 years ago

Ben's commits with the config and template image was before the apartment registration PR so these configs should be checked to make sure they work with that new feature (since it requires some extra properties to be set)

yellenlab commented 5 years ago

@eric-czech You are definitely right about new problems with the ML01 chip config after your latest update. I re-did the Via.html files in your new format and uploaded the new files. I still have to validate that they identify the correct features on the chip, and the code is currently running. Will report later on that.

eric-czech commented 5 years ago

Hey @benjaminyellen ,

I ran that multi-color dataset you shared with your new MLadder chip configuration and it looks pretty good as far as I can tell. Or at the very least, I think one way of looking at is like this where the number of images per apartment (i.e. "Number of Measurements" in the Arrays page in the app) should be 1 everywhere:

mladder_oob

There are a few apartments assigned nonsense st/apt numbers (highlighted in red circles) and a few in the middle of the array with 0 (dark blue) or 2 (green) associated images, but overall that seems great. It looks like there are about 40 obvious mistakes in that heatmap which implies an error rate around 7 bad apartments per 1000 (40 / (47 * 128) = .0066 = .7%). Nice job training that marker model!

Let me know if you're getting results like that too and if so, I think we can safely close this out.

p.s. this is with the apartment registration feature turned on (which is the default now)

eric-czech commented 5 years ago

This is interesting (@benjaminyellen ) -- here's the same thing with registration turned off:

mladder_oob_noreg

A rough count suggests there about 2.5x as many errors (~100 vs ~40) without that registration feature on so hopefully you're seeing about the same level of improvement over the previous MLadder results you had.

yellenlab commented 5 years ago

Hi @eric-czech, With the registration turned on, I can confirm that virtually all of the incorrect street/apt numbers are now caused by fabrication defects. There were a couple example of where the model incorrectly called 3s as 8s, however even these were due to poor fabrication resolution. This means, it's now on us to make better chips. :)

eric-czech commented 5 years ago

Excellent, thanks @benjaminyellen -- I'll close this guy out then.