Need Feature Extraction Methods

[ljchang]

1) Bag of Temporal filters 2) mean 3) max intensity

[jcheong0428]

In more detail:

1. Bag of Temporal filters as described in https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4034269/ -wavelet -histogram -> need to rename features

2. facial activity descriptors as described in http://ieeexplore.ieee.org/document/7423704/ -> includes: mean, median, min, max, range, standard deviation, range, various AUC etc -> include d' and d'' time derivatives.

3. mean

4. max intensity

[ljchang]

@jcheong0428: do you have ideas of how to do this based on what you've already been working on?

I think it will probably need to return a pandas.DataFrame object and find some way to automatically label the features. banks of gabor filters will break our column name consistency, unless we want to have another subclass for this use case.

We should already have mean, median, min, std implemented by default, but we will likely want to stack these horizontally, which also will break the column names.

Maybe we should have a FexFeatures Class? which subclasses the base Fex class. Not sure it will really give us anything beyond a pd.DataFrame unless we want to include any additional metadata.

What do you think?

[jcheong0428]

So I implemented a rudimentary version of BofT on branch boft_feature_extractor for now. But the problem is we need a way to slice and pivot a full-length FEX data (time, features) into something like (trial, time) per feature. It could easily be a 3 dim matrix such as as (trial, time, feature) User would also need to supply Onset/Offset times to separate the trials as well. Definitely something we should think about since it will affect how we build other analysis tools.

[Nathaniel-Haines]

I have been thinking about this too—what is the best way to include trial-level information?

With iMotions, I use the API to send information from my experiment software directly to the FACET output, so trial info is already integrated in the output. What do you all do?

If trial information is included in the output (i.e. the data that users are everything into Facet(), Openface(), etc.), we could just use methods like melt (see: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.melt.html) to transform the data to the correct format.

On Tue, Feb 6, 2018, 8:41 PM Jin Hyun Cheong notifications@github.com wrote:

So I implemented a rudimentary version of BofT on branch boft_feature_extractor for now. But the problem is we need a way to slice and pivot a full-length FEX data (time, features) into something like (trial, time) per feature. It could easily be a 3 dim matrix such as as (trial, time, feature) User would also need to supply Onset/Offset times to separate the trials as well. Definitely something we should think about since it will affect how we build other analysis tools.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cosanlab/feat/issues/5#issuecomment-363627107, or mute the thread https://github.com/notifications/unsubscribe-auth/AYbVK9rBm2hqp6IUJbuwS1W3xI5exWxSks5tSP9lgaJpZM4R4aq0 .

[Nathaniel-Haines]

everything --> entering

That was a typo

On Tue, Feb 6, 2018, 8:51 PM Nathaniel Haines nathaniel.b.haines@gmail.com wrote:

I have been thinking about this too—what is the best way to include trial-level information?

With iMotions, I use the API to send information from my experiment software directly to the FACET output, so trial info is already integrated in the output. What do you all do?

If trial information is included in the output (i.e. the data that users are everything into Facet(), Openface(), etc.), we could just use methods like melt (see: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.melt.html) to transform the data to the correct format.

On Tue, Feb 6, 2018, 8:41 PM Jin Hyun Cheong notifications@github.com wrote:

So I implemented a rudimentary version of BofT on branch boft_feature_extractor for now. But the problem is we need a way to slice and pivot a full-length FEX data (time, features) into something like (trial, time) per feature. It could easily be a 3 dim matrix such as as (trial, time, feature) User would also need to supply Onset/Offset times to separate the trials as well. Definitely something we should think about since it will affect how we build other analysis tools.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cosanlab/feat/issues/5#issuecomment-363627107, or mute the thread https://github.com/notifications/unsubscribe-auth/AYbVK9rBm2hqp6IUJbuwS1W3xI5exWxSks5tSP9lgaJpZM4R4aq0 .

[ljchang]

There are a couple of different ways to do this. For now, I think it is easier to slice it and extract features separately for each trial then stack. Pliers has a nice way to solve this. Check out their ExtractorResults and merge method and to_df method.

Another possibility is there is something like a sessions field. Nilearn has this option on some of their functions. it will iterate over sessions, which could be trials or subjects.

Will check out your code when I have time soon. Pretty exciting!

[jcheong0428]

Yup I think melting / pivoting could be the way to go. I think we could have a separate properties Fex.Conditions (?) where Trial properties (subID, Trial, Condition etc) can be added or extracted from the importer (if it exists).

[Nathaniel-Haines]

Maybe a "covariates" argument in addition to the "features" argument? Then users could input a list with all relevant fields that aren't in the default columns and we could include them in the output. We could also add covariate names to the metadata (or something like that) so that we can ignore those columns when using class methods.

I can start looking into this soon.

On Tue, Feb 6, 2018 at 9:07 PM Jin Hyun Cheong notifications@github.com wrote:

Yup I think melting / pivoting could be the way to go. I think we could have a separate properties Fex.Conditions (?) where Trial properties (subID, Trial, Condition etc) can be added or extracted from the importer (if it exists).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cosanlab/feat/issues/5#issuecomment-363631803, or mute the thread https://github.com/notifications/unsubscribe-auth/AYbVKxbI-ZJfQp1OAh0gCv6SPc4QA5wJks5tSQVRgaJpZM4R4aq0 .

[ljchang]

The features attribute is sort of open ended. The idea was to use it for different things.

1) Could be Y labels for classification/prediction

2) Could be design matrix for regression

3) could be trial/subject information.

[Nathaniel-Haines]

That makes sense. So then maybe our feature extraction methods should all implement the same slicing/melting scheme so that users can specify whether they want features by subject, trial, etc.?

On Tue, Feb 6, 2018 at 9:14 PM Luke Chang notifications@github.com wrote:

The features attribute is sort of open ended. The idea was to use it for different things.

1.

Could be Y labels for classification/prediction 2.

Could be design matrix for regression 3.

could be trial/subject information.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cosanlab/feat/issues/5#issuecomment-363633096, or mute the thread https://github.com/notifications/unsubscribe-auth/AYbVKyAj-1-H4ZRwUwtia0MYUZXSPkILks5tSQb5gaJpZM4R4aq0 .

[jcheong0428]

Well actually, since our class is a DataFrame it has melt as a method already. so we could leave it up to user to munge the data in the correct format before chugging it through feature extraction.

[Nathaniel-Haines]

True! I like that idea. Any default functionality that we can take advantage of will make it easier for users in the end.

On Tue, Feb 6, 2018 at 9:20 PM Jin Hyun Cheong notifications@github.com wrote:

Well actually, since our class is a DataFrame it has melt as a method already. so we could leave it up to user to munge the data in the correct format before chugging it through feature extraction.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cosanlab/feat/issues/5#issuecomment-363634286, or mute the thread https://github.com/notifications/unsubscribe-auth/AYbVKyidbITf52WtrgI9-OyBC938-z8zks5tSQiJgaJpZM4R4aq0 .

[ljchang]

There are pros and cons of making something like sessions/trials/subjects explicit as an attribute (FWIW, I like sessions as it is fairly ambiguous). On the one hand it's nice to have something that is built in and we can make it an iterable. Something like [x for x in Fex]. Where each iteration is a unique id in sessions field. If nothing is specified it could take the index as default so it would iterate over rows by default. On the other hand, we don't want to build in things into the Fex class that force users to use the tool in a certain way. That's the problem with most imaging software. For example, does this field make sense for the types of analyses we are already doing? Seems like it could be useful for event locked designs, but what about our passive movie watching? It could actually be useful if we had multiple movie watching sessions and we wanted to extract features separately for each one. Or subjects?

After thinking about this a bit more, I might be in favor of trying something like adding a sessions attribute to the class.

[ljchang]

@jcheong0428: can you add an example usage of feature extraction to the dropbox paper document? there is a link to it on the main README.md

[ljchang]

Also, @jcheong0428. Is this new method a "bag of temporal features"? or is it a temporal feature extraction (e.g., simply a bank of gabor filters/wavelets)? I think both versions would be helpful.

A bank of frequencies could be used as a basis set to estimate an arbitrarily shaped evoked response. A BOTF, could be used to extract signal of arbitrarily long time periods when the expected response and shape is unknown.

[jcheong0428]

It extracts the bag of temporal features as histograms of the convolved wavelets. It calls the wavelet function in the utils.py which can be used to generate banks of wavelets.

[ljchang]

Can we add just the bank of wavelet extraction too? This would likely be way more useful for more structured evoked responses like our pain studies. The stuff that Filippo and worked on used this approach. Basically, can you classify the state given a basis set of features for each AU (or emotion) using between subject classification. We used different types of algorithms (e.g., boosting, ridge, etc) to train the model.

[ljchang]

We also played with a few different ways to try and reconstruct the canonical response, but this is a much harder problem.

[Nathaniel-Haines]

After looking over the pliers package in more detail, I think that their way of dealing with feature extraction is clean, but like @ljchang said, maybe a bit much for our purposes.

A potential solution for feature extraction could be to have another attribute in the Fex class that stores extracted_features, and then each time another extraction method is called (e.g., Fex_instance.extract_mean()), we could add the results to the extracted_features attribute. We could do something like in pliers where the extracted features are stored as a list until explicitly merged, or we could merge as the user calls new extraction methods.

[ljchang]

I'm not sure about storing the features in the fex instance, but I do think this would be a good approach with a separate extractor class. Still thinking about the right design.

[ljchang]

ok, I've finished one version of what feature extract might look like. I still think we will eventually want to make it a separate class. Right now what I'm doing is creating two different types of feature extraction methods.

1. Single feature (e.g., wavelet, min, max, etc.)
2. Multiple features (e.g., multi_wavelet, summary)

most of the time we will probably want to use the multiple feature methods, plus they can be chained. Check out this one liner to summarize features separately for each trial stored as a session.

out = self.extract_multi_wavelet(min_freq=.1, max_freq=2, bank=4, mode='power').extract_summary(mean=True, min=True, max=True)

mean_f0.1_power_Negative	mean_f0.1_power_Positive	mean_f0.27_power_Negative	mean_f0.27_power_Positive	mean_f0.74_power_Negative	mean_f0.74_power_Positive	mean_f2.0_power_Negative	mean_f2.0_power_Positive	max_f0.1_power_Negative	max_f0.1_power_Positive	...	max_f2.0_power_Negative	max_f2.0_power_Positive	min_f0.1_power_Negative	min_f0.1_power_Positive	min_f0.27_power_Negative	min_f0.27_power_Positive	min_f0.74_power_Negative	min_f0.74_power_Positive	min_f2.0_power_Negative	min_f2.0_power_Positive
2461.144137	44.933343	2402.574771	44.147214	2014.221839	38.811971	512.611309	13.742162	2462.475962	44.960058	...	602.543583	16.144249	2457.869672	44.866899	2382.489025	43.731195	1912.255162	36.601319	323.215689	8.630029
5265.496315	89.540031	5118.986998	87.051765	4155.390288	70.722396	823.787103	13.891380	5267.860380	89.583811	...	917.327302	16.259144	5259.945557	89.434909	5086.586853	86.423143	4006.354103	67.682565	619.177557	9.123328
4091.137639	75.060201	3975.195407	72.955705	3212.115376	59.068061	621.914724	11.920107	4092.891901	75.091624	...	679.395476	13.247206	4087.309770	74.996833	3953.541194	72.608025	3107.787845	56.823678	531.676398	9.375654
2846.928840	17.869117	2768.593832	17.499784	2252.233890	15.029030	464.397165	5.173803	2848.161178	17.877052	...	512.429771	5.993832	2844.160927	17.850540	2752.757810	17.390787	2182.393262	14.495934	376.414411	3.500622
2498.760270	13.820603	2428.611099	13.383045	1967.403351	10.540790	384.432885	1.517818	2499.860093	13.827071	...	424.615787	1.736872	2496.192244	13.805247	2413.716199	13.292629	1899.858635	10.125063	296.667824	1.118280

This can be directly ported to analysis now using sklearn.

[ljchang]

Ok, I think this is the last thing we need to actually start using this to train models. I've added a couple of normalization methods to baseline.

Now you can iterate over all trials, extract power from a bank of wavelets, normalize each frequency band to percent signal change, and then create an overall summary of each feature (e.g., max, mean) in one line of code.

out = self.extract_multi_wavelet(min_freq=.1, max_freq=2, bank=4, mode='power').baseline(baseline='median', normalize='pct').extract_summary(mean=True,min=True,max=True)

Here's an example in action on test data.

feat = dat.loc[:,['Positive','Negative']].interpolate().extract_multi_wavelet(min_freq=.1, max_freq=2, bank=8, mode='power', ignore_sessions=True)
out = feat.baseline(baseline=feat.loc[0,:], normalize='pct', ignore_sessions=True)
summary = out.extract_summary(mean=True,min=True,max=True, ignore_sessions=True)

There is something weird going on where summaries look identical. Will investigate more.

[Nathaniel-Haines]

This is great!

I am working on a class for feature extraction, so I will incorporate your recent changes.

On Wed, Feb 14, 2018 at 10:41 AM Luke Chang notifications@github.com wrote:

Reopened #5 https://github.com/cosanlab/feat/issues/5.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cosanlab/feat/issues/5#event-1474061095, or mute the thread https://github.com/notifications/unsubscribe-auth/AYbVK8iNiJInajRY4DUQ6lCOgtTOQVeEks5tUv6IgaJpZM4R4aq0 .

[ljchang]

Ok, I think I fixed the problem, I thought it was an misplaced axis statement, but it turns out I wasn't properly passing the ignore_index keyword. This looks right now. It's pretty cool. Basically, the features that we would pass into a prediction model. Summaries of a bank of wavelets. The normalization is pretty finicky. Right now I'm using the first sample as the baseline reference. This isn't built into the baseline function yet, but could be easily.

feat = dat.loc[:,['Positive','Negative']].interpolate().extract_multi_wavelet(min_freq=.1, max_freq=2, bank=8, mode='power', ignore_sessions=True)
out = feat.baseline(baseline=feat.loc[0,:], normalize='pct', ignore_sessions=True)
summary = out.extract_summary(mean=True, min=True, max=True, ignore_sessions=True)

f,a = plt.subplots(nrows=2, figsize=(12,10))
sns.heatmap(out.T, ax=a[0])
summary.T.plot(ax=a[1],kind='bar')

[Nathaniel-Haines]

With the new Fextractor class, the .summary() method is no longer working. Is the intention of the summary method to summarize results of other feature extraction methods (e.g. summarizing wavelet extraction with mean, min, max, etc.)? If so, I wonder what you all think the best implementation would be. For example, should we add an argument to Fextractor.merge() that summarizes results before outputting a DataFrame, or should we add an argument to each of the extraction methods? I am sure there are other options too. Let me know what you think.

[jcheong0428]

Is it the case that the .summary() and Fextractor class try to achieve the goal? My understanding seems they are somewhat redundant with the difference being that the summary() method takes arguments to decide whether or not to extract/combine features, while the Fextractor class allows you to run each method then combine them.

For instance, my understanding is that to extract means you could either do summary = fex.extract_summary(mean=True) or

extractor.mean(fex_object=dat)
summary = extractor.merge(out_format="long")

Also the reason summary stopped working seems to be that you moved the methods over to Fextractor class which will prevent users from running simple methods on the Fex class unless they convert to Fextractor. Perhaps this could be prevented if Fextractor inherited the Fex class.

going back to @ljchang 's questions 1. Should we have an extractor method on Fex class that calls the new extractor class and extract methods ? Seems handy from a streamlined workflow perspective. I think the Fextractor class could just be a method on the Fex class that would function like the Fex.summary() that could be renamed to Fex.extractor(). In the end we should choose one over another. For now I think we could keep it both ways and beta test it to see which one is easier to use.

2. How should we handle applying multiple feature extraction methods? should this be input as a list of extraction methods or dictionary with variables to pass? Should we require users to just chain methods? Or should we rely on the summary method? Probably a dictionary if we want it to be most flexible. That way you could call functions that require different parameters.

3. Also should we retain the original feature extraction methods on Fex? or should they be completely moved into this new class? My vote is to either retain them or add an extractor method. I also vote to retain the original extraction methods under Fex class. I suggest we rename the summary() method to extractor() which would call the other methods. BTW I really like the Fextractor.merge() method which could also just be a method under the Fex class.

[Nathaniel-Haines]

@jcheong0428 yes I moved all the extraction methods to the Fextractor class. That said, @ljchang made some great points. If users are not interested in many different features, then the Fextractor class may be a bit of overkill.

For flexibility, I also think it would be best to retain the methods under the Fex class. That way, users will not need to worry about the Fextractor class unless they are extracting many features.

I am glad you like the merge method!

[ljchang]

I agree with @jcheong0428 that we keep both options open for now and then see which one ends up being more useful after playing with them.

I don't think it makes sense for the extract class to subclass the Fex class. The data structures should be different. I also don't think it makes sense to add a merge method to the Fex class as that will only work with a data structure that tracks multiple feature extractions. I do like the summary method that we had though. That one is really handy.

[ljchang]

I'm going to close this issue for now, but it may be worth thinking more about analysis at this level in the future.