Also feed in existing detections (if they exist) into processing pipelines

[mattdawkins]

This is to support pipelines which run object trackers from seed boxes (e.g. user draws a box, tracker tracks object, outputs new track). This can be accomplished by adding the command line argument:

-s detection_reader:file_name=[csv_filename].csv -s track_reader:file_name=[csv_filename].csv

into kwiver runner

[mattdawkins]

(they can point to the same file by the way)

[subdavis]

Questions for @mattdawkins

• How do we know if a pipeline supports this?
• Should the user have to specify whether or not to pass in existing detections?
• Could we start by always passing in detections if they exist?

[mattdawkins]

How do we know if a pipeline supports this?

We could either make a list I populate somewhere in viame, or check the .pipe for either of the above 2 processes (detection_reader or track_reader will be someone near the top of the .pipe) or

Could we start by always passing in detections if they exist?

Yes, passing these parameters in for pipelines which do not have a detection/track reader will have no effect; the main question is there any computational overhead on the web side by passing them in.

Should the user have to specify whether or not to pass in existing detections?

No, this is an un-necessary step we wouldn't want to put on the user. The main use cases are:

• User-initialized object tracking (have this in desktop now), ie user draws first box tracker tracks the rest
• Utility pipes: turn boxes to pixel masks or polygons, consolidate frame labels, others

[BryonLewis]

I have the initial version of this working for all pipelines with the desktop version. I really think we need to have a JSON format for pipelines and training and that determines if detections should be fed in.

557 discusses this and I will be templating out a format for this soon to be loaded and used to filter and display the pipelines as well as utilizing input files when needed.

[readicculus]

I want to add another few use cases this is the most relevant thing I've found to what I'm trying to do even though this is closed already.

• We have a lot of point annotations on older datasets and I'm working on providing a pipeline to automate point->box
• Pipelines that add attributes to existing tracks/detections

I'm a bit confused with the whole track/detection port stuff as it seems viame/seal-tk/dive all pass different things to the pipelines and all do different things with the returned track/detection results.

I've developed most of these pipelines/processes already but realizing in desktop viame this is not possible as any returned tracks are assigned unique ids. Ideally if I return existing track ids from a pipeline it would "update" them in the gui. Update in quotes because my understanding of dive so far is that thee csvs are somewhat immutable and it would create a new one instead of actually modifying the original. Is this correct and can I do something like this track/detection updating in DIVE currently?

[subdavis]

I'm not entirely sure I'm following. I'll try to describe in detail how DIVE handles pipeline runs right now, and maybe that will help us identify the parts that need to change to meet your use case:

• DIVE has an internal JSON format that is loaded and saved when you do manual annotation (drawing boxes).
• When a pipeline that requires those annotations as input is run, DIVE runs an export from this internal format to CSV. The export preserves all identifiers (track IDs).
• When a pipeline returns CSV annotations (either tracks or detections, there's no real difference, detentions are just tracks of length 1) that output data is read in and completely replaces whatever data was in DIVE. It's a full overwrite.
• The responsibility for performing updates is on the pipeline side. To take in 100 tracks and modify their attributes without changing their identifiers, a pipeline should simply write out the same data it read in with the appropriate changes.
• For DIVE, the VIAME csv format is only used for communication with pipelines. It exports to CSV for inputs, and it parses CSV back to its internal JSON from the response.

I'm not as familiar with seal-tk or view, but I hope that's a helpful summary.

The precise arguments to a viame pipeline (for an image sequence) from dive are below. Video is roughly the same.

source "/opt/noaa/viame/setup_viame.sh" \
  && "kwiver" runner \
  -p "/opt/noaa/viame/configs/pipelines/pipename.pipe" \
  -s input:video_filename="image-manifest.txt" \
  -s detector_writer:file_name="detector_output.csv" \
  -s track_writer:file_name="track_output.csv" \
  -s detection_reader:file_name="input.csv" \
  -s track_reader:file_name="input.csv"

Note that both track and detection readers are supplied the same file. This is by @mattdawkins's instruction.

On the output side, DIVE checks that one of the two output csv files has nonzero length at the end and simply chooses the one that isn't empty to re-ingest and overwrite the previous data.

Finally, DIVE determines whether or not to provide input to a pipe based on its name. Anything named utility_<whatever>.pipe will take input. This is just temporary until we can do something more robust.

EDIT: One other thing: As I understand it, you have annotations in some other data format that you're converting to VIAME csv with bounding boxes. I'm not really sure how DIVE fits into your use case at all, since this older point-only data will not open in DIVE.

[readicculus]

that output data is read in and completely replaces whatever data was in DIVE. It's a full overwrite.

Thanks for the response, that answers my question.

So if the UI has a bunch of tracks/detections that are points and the pipeline gets that then outputs a set of tracks with the same track_ids and other track attributes as the original points except with a box now, then the GUI would appear as if the points were turned to boxes. It's important for us to be able to relate different versions of our label datesets through consistent track_ids but seems like this is totally possible in dive and to do it in the gui instead of just a command line pipeline which I think holds value to the people who will be using these pipelines in the end.

Thanks!

[subdavis]

So if the UI has a bunch of tracks/detections that are points

See my edit: I'm not sure it's that simple.

VIAME csv (as a format) requires bounding boxes. The specification is here: https://viame.readthedocs.io/en/latest/section_links/detection_file_conversions.html

I don't know how these points would get into the UI in the first place if they didn't come from a VIAME CSV compliant annotation file. I suppose you could make it x,y,x,y or something so that the bounding box is the same as the point, but I don't know what your point-only data format looks like or how you intend to convert it to VIAME CSV. And if you have to automate point-format-to-viame-csv conversion as a pre-step to opening data in the UI, it seems like that defeats the purpose of using the UI at all.

What is the point file format you're using? It's possible that we could just support this new format directly in DIVE and allow users to open them directly. We already have the "infrastructure" for this sort of thing in client/platform/backend/serializers.

If user experience is a primary concern, perhaps supporting your annotation format as a first-class citizen in DIVE is worth discussing.

[readicculus]

yea points are just boxes x,y,x,y already in viame csv format with track_ids etc from labeling work that was done before I began working on this project(NOAA's arctic seal aerial surveys).

Good to know about the more modular serializer capabilities in dive though, I don't see us needing that for this but could see it being useful in future projects.

Right now there is just a lot of tooling I've created for myself to work with this data and am working to integrate these tools through kwiver pipelines/processes/algorithms and where possible make them usable in the guis though not crucial. Thanks again for the info and I'll definitely be in touch if thats alright as I imagine more questions will come as I dive deeper 😉 sorry, had to..