Workflow Documentation

[yuerout]

Below were questions I had regarding the general workflow of IBEIS. The answers were provided by @Erotemic Q: How to used the advanced ID interface? A: The entire idea is that you are filtering which annotations will be in the query set, which are in the database set, and what algorithm details to use. You probably don't need to worry about the last part. Just try to find the option that queries everything against everything. That will yield the most results that you can use to link which annotations are of the same individual.

Q: I tried setting the advanced ID interface as this: I assume selecting None for all the configuration keys would allow all the images I imported to be in the query set and database set? But I still can't obtain the pairwise scores [of every image against other images]. There were only 8 outputs from 14 images. A: Seeing the interface is helpful. It reminds me of what I was thinking when I designed it. Set the review config to increase ranks_top from 1 to 14. Also uncheck filter reviewed and filter true matches. That should show the entire query result rather than a subset

Note that the concept of pairwise scores doesn't make much sense here. The scores are computed with respect to an entire database, it's not guaranteed that every image will compute a score against every other image. Although in this case it probably will. But once you have thousands of images in the database that will not happen.

Also note that the scores are very much dependent on the contents of the database and the query image selected. One of the things I struggled with in my PhD was trying to find a way to automate the review process for the most confident matches. The problem is the scores simply aren't comparable nor are they separable. The manual review process is critical. The correct matches do tend to be at the top of the ranked list, but that's the only reliable behavior.

Technically IBEIS does have a concept of pairwise scores, but IIRC it's not part of the main interface. I'm not sure there's even an easy way to force that computation outside of the python API. I don't think the one versus one API was ever integrated into the GUI. It might be available through some secret right click interface though.

Q: Thank you so much for this information!

So if we have a database of images, and we have a query image that we would like to find the match for in the database, we would first import the database, add annotations, and set the species.

Since the database would contain known jaguars, would the names be set in the Name column?

After setting up the database we would like to find the best matched individual in the database for the query image, so we would import the query image and add annotation and species. How should we use the Advanced ID Interface to select only the query image?

Overall I'm a bit confused about the workflow to achieve the goal of finding best matches of the query image from the database. Also are grouping and setting exemplars necessary? If so, what do they do?

A: Wrt your first paragraph that is correct. Set the name for each known jaguar in the names column. You will then see the grouped in the names tree.

If you have just a single new query it may be faster to simply select it in the annotations table and execute a single query against the entire database (via the Q hotkey, and I think there is a menu action for it too).

The advanced ID interface is best suited for larger batch jobs. But IIRC there might be an option to query only annotations that don't have a name (the convention for a unknown name is 4 underscores: ____).

If you don't have a large database exemplars are probably unnecessary. The idea for them is you choose the best images to represent an individual so only those are used in the database. This makes it easier for the algorithm to distinguish between an individual with 2 annotations and an individual with hundreds. It keeps the number of annotations per individual in the database balanced.

[yuerout]

I set the names of the jaguars manually in the name column: Is there a way to set them more quickly (ie. import a csv with the names of the jaguar and while file they correspond to)? Then I imported a new image and added annotation and species, and used the Q hotkey to query it against the other images: The query result put the correct match at the top, and I manually reviewed it. The query image now appears in the tree of names.

Is that the correct workflow for looking for best matches? In the future, when I have jaguar images that I do not know the identity of, would I just assume that best match is the top one?

For the exemplars, does setting certain images of an individual in the database as exemplar mean that only the exemplar images will be considered when doing a query? Would it be the same for me to pre-select the best images to import to database?

Also does grouping by GPS affects the algorithm in any way?

[Erotemic]

@yuerout Thank you for compiling this. @bluemellophone @holmbergius this might be a useful resource for people working on algorithmic documentation for WBIA.

To answer the most recent set of questions:

On Importing Known Names

• Yes, there is a mechanism to import from a CSV, but it's a very specific format, and I don't immediately recall the details, but I can describe how to do it programatically if you don't mind using a little bit of Python.

First: open the GUI, and then in the "dev" tab there is an option "Launch Developer IPython", which will drop you into an embedded IPython session. The variable ibs is the currently loaded IBEIS database and gives you control over everything.

# Get a reference to every annotation
annots = ibs.annots()

# Get the image id from the annotation to get a reference to each containing image
images = ibs.images(annots.gids)

# This is a list of names corresponding to each annotation, lets say the name of each image is the name of the annotation
list_of_gnames = images.gnames

# Use the image names to compute or lookup whatever you want the name for each annotation to be
new_annot_names = []
for gname in list_of_gnames:
    new_name = pathlib.Path(gname).stem
    new_annot_names.append(new_name)

# Now you can set the names, which will write them to the database
annots.name  = new_annot_names

Then type "exit" to quit the IPython session, and you should find that the GUI is updated with your new information. (If it doesn't there is a "Refresh Tables" option in the menu). It is also possible to specify this process completely programmaticly. You can just use:

ibs = ibeis.opendb(<path-to-database>)

To get a reference to the ibs database object, and then the rest is exactly as described above. There is a lot you can do with that object if you are interested in scripting.

On Future Queries

Is that the correct workflow for looking for best matches? In the future, when I have jaguar images that I do not know the identity of, would I just assume that best match is the top one?

I think you are doing this, but for the benefit of anyone else reading this: When you find a correct match, make sure you mark is as either "True", "False", or "Incomparable" (True is the one that matters the most, you can skip False / Incomparable if you want, but it will also prevent those matches from coming up again, so it can be useful). You can do this by right clicking the row for the candidate match, and then selecting the appropriate option. (these also have hotkeys, so you can also press T or F).

When you hit "T" on a correct match it will change it's status and merge the individuals into the same "name".

When you hit "F" it will mark it as incorrect.

You will also see that some rows might be marked as "True" or "False", but be marked as "unreviewed" (indicated by a lighter red or blue color in the reviewed column)

This indicates that both the animals have the same name, but this specific pair of images doesn't have a manual review. Adding a manual review here when you are confident can help correct mistakes. (E.g. when you incorrectly mark two individuals that are not the same as belonging together, having multiple pairwise reviews can help determine where the error occurred).

Lastly, I would never assume that the top match is the correct one, but if a correct match exists, then it often will be the top one. The probability will vary by species. Here is a result from my thesis showing the probability a correct match is in the top 1, 2, 4, 5, etc.. ranks for various species:

I'd image jaguars will be similar to Grévy's Zebras in terms of precision @ rank 1, so you're looking at about 80% of the time the correct match will be rank #1 (when it exists).

On Exemplars

For the exemplars, does setting certain images of an individual in the database as exemplar mean that only the exemplar images will be considered when doing a query? Would it be the same for me to pre-select the best images to import to database?

Yes, that's exactly it.

On GPS

The only thing GPS is used for is to "Group Occurrences" (sometimes referred to as Encounters, the terminology is weird, see the Darwin Core for the correct definitions). But the idea is that an Occurrence is a bunch of images collected in a similar spacetime location. In Actions -> Group Occurrences, that will bring up a dialog that lets you set a seconds threshold (which I define as how far an average zebra can walk in that amount of time - great and totally intuitive definition I know), and that will let you group those batches of new images by GPS. You could then use those occurrences in a workflow (we used this to pre-group multiple images of the same zebra heard from different cameras), but it's not strictly necessary.

So in short, it doesn't impact the algorithm once it is running, but it can be used to select query / database annotations.

[yuerout]

Thank you for the information!!

If we want to use the one vs one interface, how should we access and use it? I checked http://erotemic.github.io/ibeis/ibeis.algo.hots.html but the one vs one module seems to be removed from the repository?

[Erotemic]

It used to be possible to right click a potential match, and there was an option for "Tune(VsOne)" but I'm currently getting an error:

  File "/home/joncrall/.pyenv/versions/3.7.10/envs/pyenv3.7.10/lib/python3.7/site-packages/plottool_ibeis/draw_func2.py", line 1637, in adjust_subplots
    del adjust_dict['validate']

that probably needs to be fixed before that interface can be used again.

There is code related to the "verification" part of my thesis in ibeis/algo/verif/vsone.py, but I don't think that is exposed in the GUI very well.

Also see ibeis/algo/verif/pairfeat.py for the PairwiseFeatureExtractor, which returns a lot of measurements between annotation pairs that can be used to train a classifier. I'll try and fix the plottool issue when I get a change (might take awhile).

[yuerout]

"Tune(VsOne)" was working for me: Is this only for potential matches? I saw that for entries with a low score in the query result, the scores show up as all 0:

{
    'sum(match_dist)'    : 0.0,
    'sum(norm_dist)'     : 0.0,
    'sum(ratio)'         : 0.0,
    'sum(ratio_score)'   : 0.0,
    'sum(sver_err_xy)'   : 0.0,
    'sum(sver_err_scale)': 0.0,
    'sum(sver_err_ori)'  : 0.0,
}

Is there a way to compute the one vs one score of every possible image pairs?

Also if we would like query against only images with a name, how should we set the advanced ID interface?

In addition, how's grouping used for selecting annotations? Is there anywhere in the advanced ID interface that allows for selection of certain groups?

[yuerout]

I noticed that in pipeline.py under request_ibeis_query_L0(ibs, qreq_, verbose=VERB_PIPELINE), there's a line

assert qreq_.qparams.pipeline_root != 'vsone', 'pipeline no longer supports vsone'

Is there any other way to use the API to compute vsone score?

[Erotemic]

There should be, but as I'm finding out a lot of code paths have broken with age. However, have a workaround that should get you want:

The pairwise features in the verif module compute one-versus-one matches. Each feature match is given a score, and we can sum those scores to get a rudimentary scalar score for each match. It won't be as good as the learning based approach, but it does give you want you want.

    >>> from ibeis.algo.verif.pairfeat import PairwiseFeatureExtractor
    >>> import ibeis
    >>> ibs = ibeis.opendb('testdb1')
    >>> extr = PairwiseFeatureExtractor(ibs)
    >>> import itertools as it
    >>> # Enumerate all possible pairs of annotation ids
    >>> all_edges = list(it.permutations(list(ibs.annots()), 2))
    >>> # Extract pairwise features for every requested pair of annotations
    >>> matches = extr._exec_pairwise_match(all_edges)
    >>> results = {}
    >>> for match in matches:
    ...     aid1 = match.annot1['aid']
    ...     aid2 = match.annot2['aid']
    ...     # Sum the fs (feature score) to get a simplified scalar score for each pair
    ...     score = match.fs.sum()
    ...     if score > 0:
    ...         results[(aid1, aid2)] = score

Replace the "testdb1" with the path to your IBEIS database (or use the exiting ibs variable when you do a developer embed).

[yuerout]

Thanks for the answer! For the pairwise feature extractor, when I look at the score of each feature match. for example, with matches[1601].fs, I got a list of scores: My understanding is that each score corresponds to a feature match, and each feature match is a region of interest on the annotation. Is this correct?

In addition to this, does the elliptical shape on the query results also correspond to feature matches? If so is there a way to extract them?

[Erotemic]

Yes, absolutely, you're starting to drill down to the core of the system. I'm glad you're able to make use of the API. Note, that I've spent the last few weeks intermittently working on getting the libraries back into working order for modern Python on Linux distros. I'm hoping to fix some of the bugs that are cropping up to make the system more usable.

But back to your question, all of the attributes are accessible in the above example via the match object. The names aren't necessarily obvious, so I'll walk you through it.

# Can access the annotations, which are "lazy dictionaries" of properties that might be in the database.
annot1 = match.annot1
annot2 = match.annot2

The match.annot1['kpts'] are the keypoints for that annotation. The format is an Nx5 array, where columns 0, and 1 are the xy coordinates and columns 2, 3, and 4 are parameters for the ellipse (see vtool_ibeis for functions on how to handle them) (these are the ellipses on each animal).

The match.annot1['vecs'] are the Nx128 SIFT features associated with each keypoint (in theory they could be neural network features, but no good implementation of that).

Those kpts and vecs are the drivers of the matching process.

The match.annot1['rchip'] is the raw image that the features were computed on. This is a cropped and rotated annotation.

These annot attributes are all lazy, so if they don't exist they will be computed. There are also ways to impact the configuration of the algorithms used. The cache is aware of these and will compute new data if the config changes.

The match.fm array itself stands for "feature matches" and is an Nx2 array where column 0 are indexes into kpts/vecs of annot1 and column 1 is similar for annot2 (the line).

The match.fs is the "feature score" array, which is an Nx1 array of scores for each feature match (color of the line).

There may be more contextual information, check the dictionary keys for data you could get from each object.

[yuerout]

Thank you for this! I'm still a bit confused about the relationship between the feature matches and the ellipse that shows up on the query. After I extract the pairwise features, their feature scores is empty (so no matches?). But in the query I saw that certain key points are still paired up.

[Erotemic]

Let me explain. Consider that we have a database full of zebra images:

Properties of Keypoints / Descriptors

For each annotation we crop out the region of the image associated with it. We take this "chip" and compute "Hessian Affine" keypoints with "SIFT" descriptors. They keypoints are the location and shape of the ellipse, and the descriptor is how we mathematically represent its visual appearance. Each annotation generates A LOT (potentially thousands) of keypoints and descriptors. If you click on an image in the annotation table and then click the "Chip VIew" window that pops up, it will show ALL of the features for a particular annotation.

If we then again click one of the keypoints (ellipses) the Chip View will show us information for that SINGLE keypoint.

Notice, the point that I clicked is highlighted in blue. All the thousands of other keypoints are colored in orange. The bottom row shows a zoomed in view of that keypoint by itself, a "normalized" patch with the a visualization of the SIFT descriptor overlaid on top of it (notice how the arrows sort of point towards brighter areas, that is how the appearance is encoded). On the right is the SIFT descriptor again, but flattened so we can see how it "mathematically looks" to the machine.

Let me know if you have more questions on keypoints and descriptors. The main point is each image generates a lot of them.

Now we have keypoints and descriptors for each annotations. Great. How do we use them? Well, the idea is that at least a few of the keypoints will land on some distinctive region of the animal. In different images of the same animal there should be a similar keypoint generated in a similar location (due to the hessian affine algorithm). And the descriptors we extract should be fairly similar to one another as well (due to the SIFT descriptor). But they won't be exactly the same. And there will also be a lot of non-distinctive parts of the image that might generate SIFT descriptors that look similar even though they are on different animals.

For example, this pattern is not very distinctive. Another animal might reasonably have a descriptor that looks very similar to this (and when I say "looks similar" I mean the L2 distance between the descriptors is low).

On the other hand, this feature is very distinctive, so it other animals will likely not have a similar pattern:

But in any case, even if we do detect the same pattern in a different image of the same animal, the SIFT descriptor won't be exactly the same, it will just be pretty close

(the above example isn't perfect, but you get the idea).

Properties of Queries

Now, there are two ways we can use these properties of the descriptors. In short we can either do a "One-vs-Many" query or a "One-vs-One" query. How we use the keypoints and descriptors will differ, which is why you see there are no matches in the vsone query, but there are matches in the vsmany query.

One-Vs-One

In this case we only consider two images. Are they a match or not? We don't have outside information. What we do is we use "Lowe's Ratio Test" to measure the distinctiveness of keypoint matches between the pair. For each feature in the first image, we find it's nearest neighbor (wrt the SIFT descriptors) in the other image and consider it a candidate match.

Now, just finding the nearest descriptor is going to return a mess (notice the ratio_thresh config is set to 1 here):

We found a match for every keypoint, and we filtered nothing! Let's use Lowe's ratio test to filter things down. (I'll set the ratio thresh to 0.8).

Now we are getting somewhere. There are still a lot of bad matches though, but there are also a lot of good ones with high scores (bright colors).

To further filter this down we can use "spatial verification" (the sv_on checkbox), which will remove spatially inconsistent matches via the RANSAC algorithm:

and now we have a reasonably good set of matching features.

One-Vs-Many

For one-vs-many queries we use the LNBNN (local naive bayes nearest neighbor) algorithm and leverage the assumption that we will have a lot of different individuals in the database. In this case we don't have Lowe's ratio test, so we can't filter via that, but we also don't have the problem that every keypoint/descriptor will generate a match to the same image. The nondistinctive ones will be spread out across the database (and it's also a lot faster to compute matches this way).

Here I get:

because my database isn't too big, there are non-distinctive random matches between the two images. Spatial verification probably removed a bunch of them

(If I turn SV off in the global Preferences, I get:

which shows that there were some spatially inconsistent matches that were filtered.

But again, we don't have the ratio test here to filter things out. Instead the filtering is implicit by the virtue of non-distinctive features mapping to effectively random images in the database.

Summary

So I hope that explains it. You don't get the same set of matches because the 1v1 and 1vM matching algorithms are fundamentally different.