spigo900
commented
2 years ago General thoughts
I think driving most of the work using a script is probably a good idea. Preprocessing has an annoying number of steps at the moment (strokes->train->decode train->decode test). So it sounds like this script would do validation on the path inputs, invoke and run preprocessing steps, then run ADAM decode? I think that makes sense. The fewer commands we have to enter live, the better. 🙂
Preprocessing
Segmentation
On segmentation, I'm not sure the easiest way, though I found a little info. I did a quick search but didn't find (probably for lack of the right terms) an obvious "plug and play" model setup. From my search though it sounds like the search term for what this model does is either "semantic segmentation" (which is by type of object, e.g. all balls in the scene are marked the same) or "instance segmentation" (which is by instance)? I don't know what we're using now but given the files are named semantic_(stuff).png I'd guess semantic segmentation. A complication is that it looks like some such models rely on a separate object detection step (to "find the bounding boxes") and that would make things more annoying.
This leads to some questions for @shengcheng and @blakeharrison-ai. IIRC @blakeharrison-ai and @shengcheng do the segmentation (for the semantic_(stuff).png images) using a model that's integrated with the simulator -- is that true? Assuming it is, how easy would it be to run the same segmentation setup/model outside of the simulator on a raw PNG image? Also, is "semantic segmentation" the right "search term" for a model that does what this step is doing, or is there more to it?
Missing strokes
I think as long as we have features.yaml, even if it's trivial, ADAM shouldn't crash when decoding. For some images we might not end up with any strokes, hence no decode and no features.yaml. As part of #1151 I'll need to fix the modified GNN code to always save at least a trivial features.yaml -- as things stand it is not doing that.
Speed
Speed-wise, I think we're okay for a small-scale demo. Stroke extraction is relatively slow and may be the slowest part right now, but probably doable for small numbers of images. On my local machine it takes ~5 seconds to extract strokes from one image. I haven't checked GNN decode speed locally yet. Hazarding a guess, I think we could handle about 50 images in ~5 minutes or 10 images in 1 minute.
Larger scales (>50 images) would be harder to do live. I don't know that we need to worry about it given there probably won't be time to go over 50 samples in detail in a demo regardless.
If necessary though I think we could solve the "lots of samples, live" issue by extracting/decoding in parallel with the display. I think we If we can do (partial) display in tandem with decode then this might be fine although we'd have to check on how workable that is. The decode saves "live," and I believe ADAM saves its decodes that way too. The first complication is ADAM would probably need to wait on stroke extraction to complete -- as is it would see "no features here" and crash. This is solvable with some trouble to plumb through the logic of configuring retry time/max retries or however we set up the coordination. The second complication is that we have to worry about missing strokes. The complication is I'm not sure if the UI can currently read it in a partially-complete state. If we don't have info.yaml or if the number doesn't match the current number of sample directories that might cause problems for the UI.
Minor issue: Curriculum name & loading
One minor issue we'll want to address is "ADAM (phase3_load_from_disk) refuses to load curriculum names we haven't predefined in the code." So we'll have to either change that or define in advance a "whatever DARPA gives us at the demo" curriculum name to run on. Either should be easy to do though.
Questions
Setting up the UI to load new data automatically -- that sounds like a good idea. Does this require changes to the backend?
When you reference new learners, I assume that's referring to the experimental & observational affordance learners?
sidharth-sundar
lichtefeld
The following is my current outline for the needed components that need to be implemented for a decode pipeline. First I've listed how I think the live demo should be structured for runtime and then below each needed component has a heading for further details.
Demo Outline
The input is a PNG image, this needs to have some form of segmentation applied and then stroke preprocessing needs to occur. To prevent needing to make multiple command line calls in the live demo my thought in the script for the demo should just invoke a subprocess for preprocessing and wait for it to complete before continuing on to ADAM decode. Once stroke preprocessing has occurred the scene can just be loaded into ADAM and decoded as normal into a specific directory for live demos. Once the 'complete' (or similar message) has appeared in the terminal I'll just refresh the available scenes in the UI and select the newly decoded scene.
Demo Script
Ideally this demo runtime environment can load a pickled ADAM learner for decoding so we can train a learner offline and not need to run any training prior to the demo. This would allow us to potentially invoke the entire pipeline for each image to be processed but I think being able to load the learner configuration once and then pass in files to process would be better for a demo. This requires:
Preprocessing
The preprocessing work is in progress for integration by @spigo900. An ideal goal is a single python script entry point for preprocessing an individual image rather than an entire curriculum at once.
Segmentation
@spigo900 Noted that the current preprocessing requires a segmentation file. We won't have that from just a PNG so how could we acquire that in the easiest way? Alternatively, can stroke extract run directly on the PNG images with success.
Decode
We'll need to have a trained ADAM learner for live decode. It may be nice to have models trained with different amounts of data to investigate any reasonable differences in decode output but mostly I think we should use the model with essentially all available training data produced used for decode.
Testing
I plan to take ~3-5 images of a chair & table from my apartment for some basic testing images. I can look through the other objects we have to see what else I could easily acquire an image of. This will hopefully allow us to test this implementation well before the live demo.
@spigo900 @sidharth-sundar @marjorief -- Any comments or concerns with this general layout?