Enforce headers on detection/prediction/classification to index columns

[as6520]

In GitLab by @cfunk1210 on Feb 19, 2021, 12:51

Currently, we have a JSON to dictate the column order but this is unnecessary when dealing with panda DataFrames since we can index them directly if they have headers. We don't need to have a separate place for headers and then ordering doesn't matter in the output (since we can identify things by column headers. This will also simplify creating the output for a detector (no need to go to the json and have a rigid ordering) and the evaluate code itself.

[as6520]

In GitLab by @as6520 on Feb 19, 2021, 12:56

@owl-uccs @rwgdrummer @abhinav_ @prijatelj Could you please take a look at this and comment on it?

[as6520]

In GitLab by @rwgdrummer on Feb 20, 2021, 12:39

Is this local concern? It sounds like Chris is proposing something specific to the detector and adapter, which does not impact the rest of the system.

That being said, a little analysis from a total system perspective is useful.

Detection file is two or three columns--(id,novelty phase probability) or (id, novelty phase probability, novelty instance probability). There is little value in adding column names but I think it fine to add them. If so, what is the propose column names in this case? ID, novel_phase, novel_instance? I estimate total system impact of five working days to adjust.

Classification, each column, except for column 0, is corresponds the label id in the ontology. It is a one to one mapping to the softmax vector from a NN. The K+1 column is the novelty class column (for AR and HWR--my understanding is that Image Classification uses column 1 for novel class). What are the column names in this case? ID,0...K-1,novel_class? or the ontology? For writers, they are 0...K-1. Thus, I think column names does not add much value here other than identifying which is the novel class column (K+1). Basically, regardless of tool used, I would want to assume that column 0 is ID, K classes is number contiguous and ordered, the K+1 class is named novel_class and either at the beginning or the end of the sequence (column 1 or column K+1). Keep in mind Pandas is not the only tool used.

Characterization in M12 has groups of columns, one set for each cluster group. For AR, we have two separate characterization files: temporal and spatial. The columns are cluster identifiers, not specific labels. We have two options with characterization: (1) One file per cluster group (as is done with spatial and temporal). For WR, the cluster groups for M18 are Writing Style, Writing Size, Pen Type, Back Ground. This probably over kill and slow! For analysis, I like things together.
(2) We add group labels to the header: id,Style,Style,Style,Size,Size,Size,Pen,Pen,Pen,BG,BG,BG ,1,2,3,1,2,3,1,2,3,1,2,3 a.png,0.1,0.1,0.8,0.2,0.1,0.7,0.0,0.0,1.0,0.99,0.01,0.0

df = pd.read_csv('characterization.csv',header=[0, 1], index_col=0)

Keep in mind that we will need to update the server and evaluation code as well. With unit test updates, the amount of change here is couple of weeks of work.

Since characterization is new and currently (under OND+A) does not have feedback for characterization, we could first apply this to characterization. The server changes for characterization is likely two days of work and eval similarly.

We can then proceed with the test and eval while iterating the rest of changes in an iterative fashion. We keep the columns in the same order so it other TA2 systems are not impacted. That is my proposal.

Other assumptions: each round, the order does not change. We can have the header in each round, but it will be dropped for all but the first round (round 0). To make things simpler for down stream eval, the server may reorder the columns to a single consistent form.

[as6520]

In GitLab by @abhi2610 on Feb 20, 2021, 12:43

And I'd push back against such overwhelming change before Phase 2. We don't want to make these many changes before M18 or ask other TA2s to make them.

[as6520]

In GitLab by @abhi2610 on Feb 20, 2021, 12:45

Clarification, TA2s can make any change they want internally to. Their system. But we'd rather keep changes to server and eval codebase minimal.

[as6520]

In GitLab by @as6520 on Feb 22, 2021, 06:53

Adding headers and retaining them in the eval code using makes it more readable. For example in the following call m_num = arm.m_num(detections[arm.novel_id], gt[1]) the column used for detection is self-explanatory since its novel_id, however without knowing about the ground truth file, I can't say what's gt[1]. Since the columns of gt files are different, this index is only applicable for activity recognition and would change if I was in another domain even though the metric computation and interpretation are the same. So if we create a novel_id header then it can be present in any column of the dataframe and it would still be used for metric computation. It is also easier to interpret which columns are being used to compute the metric.

[as6520]

In GitLab by @abhi2610 on Feb 22, 2021, 09:17

For now, we don't want to make any changes for our GT and eval code. We have enough to do already. I'll add it to the list for consideration down the road.